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Abstract 


This  report  presents  the  results  of  a  scoping  study  that  was  conducted  to  develop  a  Research  & 
Development  roadmap  for  Project  14dj,  Modelling  and  Simulation  for  Requirements  Engineering 
and  Options  Analysis.  The  purpose  of  Project  14dj  is  to  develop  a  Modelling  &  Simulation 
capability,  comprised  of  analytical  techniques  and  software  tools,  for  addressing  human  factors 
issues  commonly  encountered  by  Canadian  Forces  acquisition  projects.  This  scoping  study 
developed  a  roadmap  for  this  research  by  developing  insights  and  research  questions  from  the 
current  Canadian  Forces  procurement  process,  the  academic  and  applied  literature  on 
requirements  engineering  and  options  analysis,  and  through  expert  advice  on  how  Cognitive 
Work  Analysis  could  be  applied  to  CF  procurement.  24  research  questions  are  developed,  which 
are  structured  into  five  specific  research  proposals  for  Defence  R&D  Canada  to  consider  for 
inclusion  in  Project  14dj. 


Resume 


Ce  rapport  presente  les  resultats  d’une  etude  de  delimitation  de  projet  visant  a  elaborer  un  guide 
de  Recherche  et  Developpement  pour  le  Projet  14dj,  Modelisation  et  simulation  pour  Vingenierie 
des  besoins  et  V analyse  des  options.  Ce  projet  a  pour  but  de  developper  une  capacite  de 
Modelisation  et  simulation  composee  de  techniques  d’analyse  et  d’outils  logiciels,  pour  resoudre 
des  questions  liees  aux  facteurs  humains  qui  se  posent  souvent  dans  les  processus  d’acquisition 
des  Forces  canadiennes.  La  presente  etude  de  delimitation  de  projet  a  permis  d’elaborer  un  guide 
pour  cette  recherche  en  developpant  des  perspectives  et  des  questions  de  recherche  a  partir  du 
processus  d’acquisition  actuel  des  Forces  canadiennes,  de  la  litterature  didactique  et  des  etudes 
appliquees  a  l’ingenierie  des  besoins  et  l’analyse  des  options,  et  par  des  conseils  eclaires  sur  la 
maniere  dont  l’Analyse  du  travail  cognitif  peut  etre  appliquee  aux  acquisitions  des  FC.  Vingt- 
quatre  questions  de  recherche  sont  developpees  et  reparties  en  cinq  projets  de  recherche  que  R&D 
pour  la  defense  Canada  pourra  examiner  et  inclure  dans  le  Projet  14dj. 
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Executive  summary 


Modelling  &  Simulation  for  Requirements  Engineering  and 
Options  Analysis:  Final  Report 

Gerard  Torenvliet;  Antony  Hilliard;  Catherine  M.  Burns;  Gavan  Lintern;  Jean- 
Yves  Lamarre;  DRDC  Toronto  CR  2010-049;  Defence  R&D  Canada  -  Toronto; 
May  2010. 

Introduction  or  background:  Defence  R&D  Canada  -  Toronto  has  recently  launched  Project 
14dj,  titled  “Modelling  and  Simulation  for  Requirements  Engineering  and  Options  Analysis”. 
This  project  is  an  Applied  Research  Project  to  develop  a  Modelling  &  Simulation  framework  and 
associated  software  tools  for  supporting  requirements  engineering  and  options  analysis.  A 
scoping  study  was  conducted  to  ensure  that  the  research  of  Project  14dj  is  properly  situated  within 
the  context  of  Canadian  Forces  procurement,  current  trends  in  requirements  engineering  and 
options  analysis,  and  the  area  of  practice  of  Cognitive  Work  Analysis. 

Results:  The  scoping  study  developed  24  research  questions  that  have  been  structured  into  five 
research  proposals  for  inclusion  in  Project  14dj:  (1)  research  to  apply  Cognitive  Work  Analysis 
and  Modelling  &  Simulation  to  the  development  of  operational  requirements;  (2)  research  to 
conduct  a  cognitive  task  analysis  of  requirements  engineering  and  options  analysis  in  Canadian 
Forces  procurement;  (3)  research  to  develop  a  tool  to  support  the  application  of  Cognitive  Work 
Analysis  to  Canadian  Forces  procurement;  (4)  research  to  extend  and  apply  Social  Organization 
and  Cooperation  Analysis  (a  lesser-developed  area  of  Cognitive  Work  Analysis)  to  Canadian 
Forces  procurement;  and,  (5)  research  to  extend  Defence  R&D  Canada  -  Toronto’s  crewing 
effectiveness  task  network  model. 

Significance:  The  research  program  presented  in  this  report  should  provide  Defence  R&D 
Canada  with  a  stronger  ability  to  have  a  positive  impact  on  Canadian  Forces  procurement 
projects.  If  successful,  this  research  could  provide  the  CF  with  an  overall  reduction  of  risk  in  the 
procurement  cycle.  This  should  lead  to  more  predictable  procurement  projects  that  are  better  able 
to  provide  the  CF  with  the  capabilities  required  to  meet  their  strategic  objectives. 

Future  plans:  The  research  proposals  contained  in  this  report  should  be  structured  into  an  overall 
research  plan  for  Project  14dj. 
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Modelling  &  Simulation  for  Requirements  Engineering  and 
Options  Analysis:  Final  Report 

Gerard  Torenvliet;  Antony  Hilliard;  Catherine  M.  Burns;  Gavan  Lintern;  Jean- 
Yves  Lamarre;  DRDC  Toronto  CR  2010-049;  R  &  D  pour  la  defense  Canada  - 
Toronto;  Mai  2010. 

Introduction  ou  contexte:  Recherche  et  developpement  pour  la  defense  Canada  -  Toronto  a 
recemment  lance  le  Projet  14dj,  intitule  «Modelisation  et  simulation  pour  l’ingenierie  des  besoins 
et  1’ analyse  des  options)).  11  s’agit  d’un  projet  de  recherches  appliquees  visant  a  developper  un 
cadre  de  modelisation  et  simulation  et  des  outils  logiciels  associes  en  vue  d’appuyer  l’ingenierie 
des  besoins  et  l’analyse  des  options.  On  a  mene  une  etude  de  delimitation  de  projet  afm  de 
verifier  que  la  recherche  du  Projet  14dj  s’inscrit  bien  dans  le  contexte  des  acquisitions  des  Forces 
canadiennes,  des  tendances  actuelles  en  matiere  d’ingenierie  des  besoins  et  d’analyse  des  options, 
et  du  domaine  d’exercice  de  l’Analyse  du  travail  cognitif. 

Resultats:  L’ etude  de  delimitation  de  projet  a  developpe  24  questions  qui  ont  ete  reparties  en 
cinq  projets  de  recherche  a  inclure  dans  le  Projet  14dj:  (1)  des  recherches  pour  developper  les 
besoins  operationnels  a  l’aide  de  l’Analyse  du  travail  cognitif  et  de  la  modelisation  et  simulation; 
(2)  des  recherches  pour  mener  une  analyse  du  travail  cognitif  lie  a  l’ingenierie  des  besoins  et 
l’analyse  des  options  dans  les  acquisitions  des  Forces  canadiennes;  (3)  des  recherches  pour 
elaborer  un  outil  en  vue  d’appuyer  l’application  de  l’Analyse  du  travail  cognitif  aux  acquisitions 
des  FC;  (4)  des  recherches  permettant  d’etendre  et  d’appliquer  1’ Organisation  sociale  et  l’Analyse 
de  la  cooperation  (un  domaine  moins  developpe  de  l’Analyse  du  travail  cognitif)  aux  acquisitions 
des  FC,  et  (5)  des  recherches  pour  etendre  le  modele  de  reseau  de  taches  de  R&D  pour  la  defense 
Canada  -  Toronto  de  fag  on  a  ameliorer  l’efficacite  du  personnel. 

Portee:  Le  programme  de  recherche  presente  dans  ce  rapport  fournira  a  Recherche  et 
developpement  pour  la  Defense  Canada  une  plus  grande  capacite  d’exercer  une  influence  positive 
sur  les  processus  d’acquisition  des  FC.  Si  cette  recherche  est  fructueuse,  elle  pourrait  permettre 
aux  FC  de  reduire  les  risques  dans  l’ensemble  du  cycle  d’acquisition.  Cela  devrait  mener  a  des 
projets  d’acquisition  plus  previsibles  qui  sont  plus  en  mesure  de  foumir  aux  FC  les  capacites  dont 
elles  ont  besoin  pour  atteindre  leurs  objectifs  strategiques. 

Recherches  futures:  Les  projets  de  recherche  contenus  dans  le  present  rapport  devraient  etre 
repartis  dans  un  plan  de  recherche  global  pour  le  Projet  14dj. 
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1  Introduction 


1.1  Background 

To  meet  its  strategic  objectives,  the  Canadian  Forces  (CF)  must  have  access  to  equipment 
appropriate  to  its  current  and  future  missions.  To  support  this,  the  Chief  of  Force  Development 
(CFD)  has  enacted  a  strategic  planning  process  called  Capability  Based  Planning  (CBP).  CBP  is 
“an  integrated  process  that  is  coherent  and  logical  to  determine  the  kinds  of  capabilities  [the  CF] 
will  field  in  the  years  to  come”  (Chief  of  Force  Development,  2008a,  p.  1).  The  process  develops 
a  prioritized  list  of  procurement  projects  which  is  updated  regularly  based  on  changes  in  the 
strategic  environment.  If  a  project  is  selected  for  procurement,  the  High-Level  Mandatory 
Capabilities  (HLMCs)  identified  in  the  CBP  process  are  passed  through  successive  operational 
and  technical  phases  of  analysis  to  capture  the  requirements  to  be  fulfilled  by  new  or  modernized 
equipment.  This  is  a  complex  process  of  translation  (from  strategic  to  operational  to  technical 
requirements)  and  trade-off  (between  operational  needs  and  what  is  available  in  the  marketplace). 

In  the  majority  of  cases,  there  is  a  mismatch  between  the  CF’s  operational  needs  and  the 
equipment  available  in  the  marketplace.  Instead  of  being  able  to  procure  Commercial-Off-The- 
Shelf  (COTS)  equipment,  the  Crown  frequently  needs  to  work  with  industry  to  develop  a  bespoke 
solution.  For  example,  the  in-development  Maritime  Helicopter  Project  is  nominally  based  on  the 
Sikorsky  S-92,  but  comes  with  airframe  enhancements,  cockpit  instrumentation,  and  mission 
systems  highly  customized  for  the  program’s  specific  requirements. 

In  these  cases,  the  Crown  is  faced  with  the  difficult  and  complex  job  of  defining  operational  and 
technical  requirements  for  a  system  that  does  not  yet  exist.  These  requirements  need  to  be 
developed  through  an  understanding  of  the  ways  in  which  operators  will  use  the  new  equipment, 
but  at  a  point  when  the  complexities  of  the  system  and  its  operational  employment  are  not  fully 
understood.  What  is  more,  new  systems  introduce  new  possibilities  for  human  work  and  so 
change  how  people  work,  usually  in  ways  that  cannot  be  predicted  a  priori.  As  a  result,  systems 
designers  get  caught  in  what  has  been  called  the  task-artefact  cycle  (Carroll,  Kellogg,  &  Rosson, 
1991):  current  tasks  are  the  basis  for  requirements;  requirements  drive  the  design  of  new 
artefacts;  the  new  designs  open  up  new  possibilities  for  work;  and  the  original  requirements  end 
up  missing  their  target. 
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Figure  1:  The  task-artefact  cycle  (adapted  from  Carroll,  et  al.,  1991). 


1.2  New  directions 

Work  to  improve  the  development  of  operational  and  technical  requirements  is  progressing  on 
two  fronts.  First,  the  systems  engineering  community  is  developing  techniques  to  improve  the 
quality  of  requirements.  Many  different  techniques  have  been  proposed  (a  sampling  of  which  is 
reviewed  in  Section  2.3),  toward  the  overarching  aim  of  producing  individual  requirements  that 
are  not  prone  to  misinterpretation  and  sets  of  requirements  that  are  comprehensive  (Halligan, 
undated).  This  work,  if  successful,  should  improve  the  translation  of  requirements  from  the 
Crown  to  vendors. 

Second,  the  cognitive  engineering  community  is  developing  methods  and  frameworks  to  resolve 
the  task-artefact  cycle.  One  such  framework  is  Cognitive  Work  Analysis  (CWA;  Rasmussen, 
Pejtersen,  &  Goodstein,  1994;  Vicente,  1999;  Vicente  &  Rasmussen,  1992),  a  type  of  analysis 
that,  instead  of  describing  current  operator  tasks,  seeks  to  identify  technological  and 
organizational  constraints  that  need  to  be  satisfied  for  a  system  to  be  effective.  The  requirements 
derived  from  CWA  are  more  fundamental  than  those  developed  from  task-based  approaches,  and 
so  systems  derived  from  these  requirements  should  have  fewer  elements  that  get  caught  in  the 
task-artefact  cycle. 


10 
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1.3  Project  14dj  -  Modelling  and  simulation  for  requirements 
engineering  and  options  analysis 

Defence  Research  &  Development  Canada  (DRDC)  Toronto  has  recently  launched  Project  14dj, 
titled  “Modelling  and  Simulation  for  Requirements  Engineering  and  Options  Analysis”.  This 
project  is  an  Applied  Research  Project  (ARP)  to  develop  a  Modelling  &  Simulation  (M&S) 
framework  and  associated  software  tools  for  supporting  requirements  engineering  and  options 
analysis.  While  DRDC  has  done  significant  work  in  this  area  in  the  past,  they  are  now  interested 
in  supplementing  their  current  toolset  with  the  constraint-based  perspective  provided  by  CWA. 
The  goal  of  the  ARP  is  to  provide  the  CF  with  a  comprehensive  and  hybrid  M&S  approach  that 
includes  constraint -based,  task-based,  and  process-based  techniques.1 

1 .4  Current  work  -  Development  of  an  R&D  roadmap 

The  purpose  of  the  work  described  in  this  report  is  to  develop  a  scope  for  the  ARP,  and  to 
formulate  this  scope  as  a  Research  &  Development  (R&D)  roadmap.  The  R&D  roadmap  is 
intended  to  define  the  research  questions  that  need  to  be  answered  by  the  ARP,  and  the  research 
methods  to  address  those  questions  appropriately  within  the  ARP’s  three-year  time  horizon.  The 
research  to  develop  this  roadmap  is  documented  in  this  report,  and  focused  on  two  issues: 

•  The  feasibility  of  applying  CWA  to  requirements  engineering  in  the  CF  procurement 
context.  To  develop  a  roadmap  for  the  ARP,  it  was  important  first  to  understand  if  and  how 
CWA  can  be  applied  to  requirements  engineering  in  a  military  procurement  context.  This 
involved  gaining  an  understanding  of  the  type  of  constraints  that  can  be  captured  by  CWA 
that  are  meaningful  for  CF  acquisitions,  the  types  of  performance  metrics  that  can  be 
developed  for  those  constraints,  and  the  prioritization  among  those  constraints.  Further,  if 
this  research  is  also  to  assist  the  CF  in  options  analysis,  an  understanding  is  needed  of  the 
methods  that  can  be  used  to  measure  the  ways  in  which  different  design  alternatives  manage 
the  constraint-space  of  a  work  domain,  and  the  advantages  or  disadvantages  of  them  as 
compared  to  existing  task-  or  process-based  methods.  Finally,  since  CWA  is  widely 
considered  to  be  a  challenging  analytical  framework,  it  is  important  to  understand  if  it  is 
possible  to  generalize  and  standardize  its  modelling  techniques  and  to  package  them  into 
software  tools. 

•  The  feasibility  of  developing  an  integrated  modelling  platform.  Since  the  objective  of 
this  research  is  to  investigate  the  integration  of  CWA  with  existing  approaches,  this  scoping 
study  also  investigated  if  it  would  be  necessary,  desirable,  and  feasible  to  integrate  these 
techniques  into  a  common  platform,  and  further  to  investigate  if  that  could  be  done  while 
ensuring  the  traceability  of  requirements  back  to  their  source  in  analysis  results. 

The  objective,  context,  and  expected  benefits  of  this  project  and  of  Project  14dj  are  summarized 
in  Figure  2,  below. 


1  Constraint -based  work  analysis  techniques  model  the  constraints  of  a  work  domain  that  must  be  respected 
for  the  system  to  operate  without  faults;  task-based  techniques  model  the  normative  tasks  that  should  be 
performed  to  work  successfully  in  the  domain;  and  process-based  techniques  model  the  processes  that 
occur  in  the  work  domain.  There  is  a  crisp  distinction  between  constraint -based  techniques  and  task-  and 
process-based  modeling,  but  there  is  only  a  fuzzy  distinction  between  task-  and  process-based  modeling. 
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Project  objective.  To  assist  DRDC  in  developing  a  Research  &  Development  roadmap  for  the 
three-year  Applied  Research  Project,  Modelling  and  Simulation  for  Requirements  Engineering 
and  Options  Analysis. 

Applied  Research  Project  objective.  To  develop  a  Modelling  &  Simulation  capability, 
comprised  of  analytical  techniques  and  software  tools,  for  addressing  human  factors  issues 
commonly  encountered  in  Canadian  Forces  acquisition  projects. 

Applied  context.  Canadian  Forces  procurement;  specifically,  the  challenges  involved  in 
developing  comprehensive,  meaningful,  understandable,  and  traceable  operational 
requirements  in  the  early  stages  of  procurement;  in  translating  operational  requirements  into 
technical  requirements;  and  in  developing  measurable  specifications. 

Benefits.  Successfully  meeting  the  Applied  Research  Project’s  objective  within  the  context  of 
Canadian  Forces  procurement  is  expected  to  lead  to  a  reduction  of  risk  in  the  procurement 
cycle.  This  will,  in  turn,  lead  to  more  predictable  procurement  projects  that  are  better  able  to 
provide  the  CF  with  the  capabilities  required  to  meet  their  strategic  objectives. 


Figure  2:  The  objective,  context,  and  expected  benefits  of  this  project  and  of  Project  14dj. 


1 .5  Approach 

The  project  Statement  of  Work  (SOW)  structured  the  work  to  produce  an  R&D  roadmap  around  a 
series  of  reviews  -  of  the  CF  procurement  process,  of  requirements  engineering  and  options 
analysis  practice,  and  of  CWA.  We  assigned  these  reviews  to  team  members  with  relevant 
expertise,  and  then  conducted  a  two-day  workshop  to  discuss  and  consolidate  the  results  of  these 
reviews  into  an  outline  for  the  R&D  roadmap.  All  team  members  participated  in  the  workshop, 
along  with  the  DRDC  scientists  responsible  for  Project  14dj.  After  this  workshop,  the  team’s 
project  engineer  worked  from  the  guidance  developed  at  the  workshop  to  prepare  a  draft  R&D 
roadmap.  The  draft  roadmap  was  revised  through  reviews  by  the  team’s  various  experts  to 
produce  the  finalized  R&D  roadmap  presented  in  this  document. 

The  level  of  expertise  brought  to  bear  on  the  development  of  this  R&D  roadmap  should  be 
emphasized;  while  the  resources  allocated  to  this  project  allowed  only  for  high-level  reviews  in 
each  area,  the  experts  consulted  had  a  strong  grasp  of  each  of  the  review  areas  involved  in  this 
project.  Our  team  included: 

•  an  ex -Air  Force  Lieutenant  Colonel  with  a  long  career  of  experience  in  CF  procurement, 
both  as  an  aerospace  engineer  in  the  CF  and  as  a  military  program  manager  at  a  major 
Canadian  defence  contractor; 

•  a  professor  in  Systems  Design  engineering  who  is  a  global  expert  in  human  factors  and 
CWA;  and, 
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•  an  expert  in  CWA  who  has  held  the  post  of  Chief  Scientist  at  a  major  US  defence 
contractor,  and  who  has  a  long  career  of  experience  in  human  factors  engineering  and  design 
and  systems  engineering,  both  from  academic  and  industry  perspectives. 

In  addition  to  this,  the  team’s  project  engineer  has  collaborated  with  DRDC  on  numerous  research 
contracts,  has  a  strong  knowledge  of  CWA,  and  has  led  the  human  factors  effort  for  an  element  of 
a  major  CF  procurement  project  on  behalf  of  a  major  Canadian  defence  contractor.  The  team’s 
research  assistant,  a  doctoral  candidate  in  Mechanical  &  Industrial  Engineering,  also  has  a  strong 
knowledge  of  CWA. 

To  ensure  that  this  scoping  study  was  properly  informed  by  the  DRDC  context,  this  project  has 
also  been  conducted  as  a  strong  collaboration  between  the  project  team  and  the  scientists  at 
DRDC  Toronto  involved  with  Project  14dj. 

1.6  Document  organization 

This  document  is  organized  as  follows: 

•  Section  1  -  Introduction  (this  section)  provides  the  project  background,  puipose,  and  the 
approach  taken; 

•  Section  2  -  Project  context  provides  the  results  of  the  literature  reviews  conducted  to 
develop  the  context  for  the  R&D  roadmap; 

•  Section  3  -  Research  opportunities  present  the  research  opportunities  for  DRDC  that 
follow  from  the  literature  reviews  and  DRDC’s  research  context;  and 

•  Section  4  -  Conclusions  provides  concluding  material. 

This  report  also  includes  the  following  annexes: 

•  Annex  A  -  Literature  review  of  current  requirements  engineering  practice  contains  the 
slides  prepared  to  summarize  the  results  of  the  literature  review  of  requirements  engineering 
and  options  analysis;  and, 

•  Annex  B  -  An  overview  of  the  CF  procurement  process  contains  the  slides  prepared  to 
summarize  the  results  of  the  review  of  the  CF  procurement  process. 
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2  Project  context 


2.1  Introduction 

As  discussed  in  the  introduction,  our  work  to  develop  an  R&D  roadmap  for  Project  14dj, 
“Modelling  and  Simulation  for  Requirements  Engineering  and  Options  Analysis”  began  by 
conducting  a  series  of  reviews  to  develop  the  information  necessary  to  properly  situate  the  R&D 
roadmap.  We  conducted: 

•  a  review  of  the  CF  procurement  process,  to  provide  information  to  help  ensure  that  the  R&D 
roadmap  is  targeted  at  relevant  opportunities  in  the  procurement  process  that  will  deliver 
benefits  to  the  CF;  and, 

•  a  review  of  research  and  practice  in  requirements  engineering  and  options  analysis,  to 
provide  information  to  ensure  that  the  R&D  roadmap  addresses  important  questions  in  the 
Requirements  Engineering  community  and  does  not  duplicate  prior  research. 

In  addition  to  this,  we  also  combined  the  team’s  expertise  in  CWA  to  develop  a  series  of  thoughts 
about  how  the  analysis  structure  and  models  from  each  of  the  five  phases  of  CWA  can  be 
integrated  into  requirements  engineering  in  general,  and  specifically  into  the  CF  acquisition 
process. 

The  results  of  the  reviews  are  documented  in  Annex  A  (Literature  review  of  current  requirements 
engineering  practice)  and  Annex  B  (An  overview  of  the  CF  procurement  process);  the  purpose  of 
this  section  is  to  present  some  of  the  most  relevant  findings  from  these  reviews  and  our  team’s 
discussions  about  CWA  and  to  discuss  their  implications  for  the  development  of  an  R&D 
roadmap. 

2.2  Canadian  Forces  procurement 

2.2.1  General 

The  section  provides  an  overview  of  the  CF  procurement  process,  for  the  purpose  of  identifying 

challenges  and  opportunities  within  that  process  that  could  be  addressed  by  Human  M&S 
techniques,  especially  as  applied  to  questions  relevant  to  Human  Factors  (HF)  and  Human 
Systems  Integration  (HSI).  This  overview  is  intended  only  to  provide  adequate  context  to  the 
challenges  and  opportunities  identified  that  motivate  specific  items  in  the  proposed  R&D 
roadmap  (contained  in  Section  3).  The  references  cited  provide  more  detailed  context  and  should 
be  referred  to  as  appropriate  when  specific  research  directions  are  engaged. 

2.2.2  High-level  overview  of  the  CF  procurement  process 

A  high-level  overview  of  the  CF  procurement  process  from  the  perspective  of  operational  and 
technical  requirements  generation  is  shown  in  Figure  3,  below.  This  figure  shows  the  major 
phases  in  the  procurement  process  and  the  typical  documentation  resulting  from  each  phase. 
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Before  describing  each  phase  of  this  process,  a  number  of  things  about  the  process  in  general 
should  be  noted: 

•  The  process  is  idealized.  As  with  all  processes,  the  documented  version  of  the  CF 
procurement  process  is  somewhat  idealized,  and  every  procurement  project  does  not  follow 
it  in  the  same  way.  Some  projects  may  follow  each  step  in  the  process  with  substantial 
rigour  and  analysis  (for  example,  the  current  Joint  Unmanned  Surveillance  Target 
Acquisition  System  (JUSTAS)  Medium  Altitude  Long  Endurance  (MALE)  Unmanned 
Aerial  Vehicle  (UAV)  procurement  project),  while  other  projects  move  from  an  identified 
capability  deficiency  to  fielded  equipment  very  quickly  (for  example,  the  CC-177 
Globemaster  procurement  project).  Nonetheless,  the  process  illustrated  in  Figure  3 
represents  how  the  CF  aims  to  run  procurement  projects,  and  so  is  a  useful  point-of- 
departure  for  the  current  work. 

•  Investment  and  funding  considerations  are  not  depicted.  The  process  as  depicted  only 
shows  the  flow  of  requirements,  from  the  identification  of  capability  deficiencies  at  the 
strategic  level  through  to  the  implementation  of  technical  requirements  and  specifications  at 
the  implementation  level.  Parallel  with  this  is  the  equally  important  process  of  developing 
an  investment  plan  and  finding  funding  for  procurement  projects.  It  is  our  understanding, 
however,  that  the  issues  related  to  investment  fall  outside  of  the  scope  of  Project  14dj. 

•  There  are  formal  interfaces  between  each  phase.  As  shown  by  the  list  of  procurement 
documents  listed  at  the  left  side  of  Figure  3,  the  formal  interface  between  each  phase  of  the 
process  consists  of  written  documents.  Increasingly,  this  documentation  also  includes 
models  developed  within  the  Department  of  National  Defence  (DND)/CF  Architectural 
Framework  (DNDAF;  see  Director  Enterprise  Architecture,  2009,  for  more  details). 

•  There  are  soft  interfaces  between  each  phase.  As  signified  by  the  overlap  between  the 
three  core  phases  (capability-based  planning,  operational  requirements  development,  and 
technical  requirements  development),  there  are  interfaces  between  these  phases  that  assist  in 
the  hand-over  of  information.  This  overlap  can  help  to  clarify  any  ambiguities  in  the 
procurement  documents  that  form  the  formal  interface  between  the  phases. 

•  There  should  ideally  be  feedback  and  recalibration  between  phases,  both  up  and  down 
the  chain.  Decisions  made  at  lower  levels  of  the  procurement  process  (for  example,  a 
decision  made  in  the  technical  requirements  development  phase  to,  for  funding  reasons, 
relax  a  specific  technical  requirement)  should  ideally  be  linked  to  their  impact  on  higher 
levels  of  the  process  (the  impact  of  relaxing  a  technical  requirement  on  the  project’s  ability 
to  satisfy  the  capability  deficiencies  it  is  intended  to  resolve). 

•  Procurement  projects  are  under-staffed  and  under-resourced.  Procurement  projects 
within  the  CF  are  perceived  to  be  chronically  under-staffed  and  under-resourced.  For  this 
reason,  the  current  focus  in  CF  procurement  does  not  seem  to  be  on  performing  extensive 
analysis  (as  this  could  lead  to  so-called  ‘analysis  paralysis’),  but  is  rather  on  reducing 
program  risk. 

Decision  making  relies  heavily  on  military  judgment.  Many  decisions  need  to  be  made 
over  the  course  of  the  procurement  process.  While  the  procurement  process  makes 
allowance  for  the  use  of  structured  analysis  tools  and  methods,  there  seems  to  be  a  strong 
default  reliance  on  using  what  is  termed  the  best  military  judgment  to  make  decisions.  The 
following  statement  in  the  Strategic  Capability  Roadmap  (Chief  of  Force  Development, 
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2008b)  is  typical:  “The  last  step  .  .  .  was  to  apply  best  military  judgment  to  refine  the  set  of 
selected  alternatives.  Every  mathematical  model  is  limited  by  the  factors  it  can  take  into 
account  to  produce  the  best  objective  solution,  (p.  52,  emphasis  ours).” 

•  Schedules  are  difficult  to  predict.  Subject  matter  experts’  experience  with  the  procurement 
process  is  that  the  schedule  for  each  phase  can  be  difficult  to  predict,  and  is  rarely  set  in 
stone.  Each  phase  may  progress  for  long  periods  of  time  with  little  perceived  scheduling 
pressure,  but  this  situation  can  change  rapidly  if  the  priority  of  a  program  changes.  For  this 
reason,  CF  personnel  are  loath  to  involve  any  outside  organization  in  the  requirements 
development  process,  especially  if  that  outside  organization’s  efforts  will  be  on  the  critical 
path. 

Each  of  these  phases  is  described  in  more  detail  in  the  sections  that  follow. 

2.2.3  Policy  context:  Ministerial  direction  and  CF  long-term  objectives 

The  CF  reports  to  the  Government  of  Canada  and  all  activities  of  the  CF  must  be  made  in 
response  to  government  policy.  Under  most  circumstances,  government  direction  is  expressed  as 
a  set  of  long-term  objectives  for  the  CF,  for  example,  the  Canada  First  Defence  Strategy  (Chief 
of  Defence  Staff,  2008).  Government  direction  may  also  come  more  directly  as  input  from  the 
Minister  of  National  Defence.  These  various  forms  of  direction  form  the  policy  context  for 
procurement. 

CF  staff  developing  operational  or  technical  requirements  during  later  stages  of  procurement  may 
feel  removed  from  this  high-level  policy  context,  but  it  is  nonetheless  important  in  terms  of 
developing  priorities  and  ensuring  adequate  funding  for  procurement.  For  example,  the  Canada 
First  Defence  Strategy  sets  a  policy  direction  for  the  CF  to  be  modernized  through  the 
procurement  of  at  least  five  new  types  of  equipment:  destroyers  and  frigates,  fixed  wing  search 
and  rescue  aircraft,  next-generation  fighter  aircraft,  maritime  patrol  aircraft,  and  land  combat 
vehicles  and  systems.  Should  the  policy  direction  from  the  government  change,  procurement 
projects  that  have  progressed  even  to  late  stages  may  be  reworked  or  even  cancelled  (for  example, 
the  Sea  King  replacement  project,  started  in  1986,  was  cancelled  in  1993  after  progressing  to  the 
implementation  phase).  So,  while  the  policy  context  may  not  affect  the  day-to-day  work  of 
developing  operational  and  technical  requirements,  it  is  the  trump  card  that  determines  whether  or 
not  the  subsequent  phases  progress  at  all. 

2.2.4  Capability-based  planning 

CBP  is  a  high-level  planning  activity  that  is  performed  in  response  to  the  current  policy  context, 
with  the  aim  of  developing  a  prioritized  list  of  capabilities2  for  procurement  over  the  current 
planning  horizon  (which  is  typically  on  the  order  of  20  years).  The  long  planning  horizon 


2  A  capability  is  defined  as,  “A  particular  ability  that  contributes  to  the  production  of  a  desired  effect  in  a 
given  environment  within  a  specified  time  and  the  sustainment  of  that  effect  for  a  specified  period. 
Capability  is  delivered  by  an  appropriate  combination  of  PRICIE  (Personnel/Leadership/Individual 
Training,  Research  and  Development/Operational  Research,  Infrastructure,  Environment  and  Organization, 
Concepts,  Doctrine,  Collective  Training,  Information  Management  &  Technology  &  Equipment  Support) 
components  (Chief  of  Force  Development,  2008a,  p.  3).” 
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involved  in  CBP  means  that  the  process  involves  developing  an  understanding  of  the  implications 
of  current  government  policy  on  the  future  security  context.  CBP  builds  this  understanding  by 
developing  scenarios  to  capture  assumptions  about  the  future  security  context,  understanding 
what  the  response  of  the  CF  to  those  scenarios  should  be,  and  abstracting  what  the  CF’s  current 
capability  deficiencies  are  with  respect  to  those  scenarios.  These  deficiencies  point  to  capabilities 
that  should  be  procured.  Since  there  is  never  enough  money,  people,  or  time  for  the  CF  to  address 
every  capability  deficiency,  CBP  also  includes  processes  for  prioritizing  the  capabilities  that  the 
CF  needs  to  acquire. 

The  most  important  output  of  the  CBP  process  is  the  Strategic  Capability  Roadmap  (SCR)  (Chief 
of  Force  Development,  2008b).  The  SCR  is  a  comprehensive  document  that  captures  all  of  the 
results  of  the  CBP  process  to  “provide  rigour  and  logic  to  planning  for  future  CF  capabilities” 
(Chief  of  Force  Development,  2008b,  p.  iii).  It  includes  documentation  of:  the  future  security 
context  and  the  way  the  CF  is  envisioned  to  respond  to  that  environment;  the  problem  space  of 
capability  deficiencies;  the  solution  space  of  capability  alternatives;  the  integration  space  of 
prioritized  capabilities  for  future  development;  important  development  risks  that  could  delay  the 
closure  of  deficiencies,  and  recommendations  for  developing  an  investment  plan  to  provide  the 
capabilities  required.  In  terms  of  providing  a  robust  and  well-considered  direction  for  future 
procurement  projects,  the  SCR  is  impressive  in  scope  and  detail.  The  most  recent  version  of  the 
SCR  at  the  time  of  writing  includes  a  prioritized  list  of  319  capabilities,  ranging  from  the 
Canadian  Surface  Combatant  (priorities  one  and  two)  to  a  personal  anti-armour  weapon  system 
(priority  319). 

2.2.5  Operational  requirements  development 

If  a  project  identified  in  the  CBP  is  given  preliminary  funding,3  the  project  is  allocated  to  one  of 
the  environments  (Army,  Navy,  or  Air  Force)  and  a  group  of  operators4  are  assigned  the 
responsibility  of  using  their  knowledge  to  develop  an  operational  vision  and  requirements  for  the 
capability.  While  we  were  not  able  to  locate  documentation  about  the  work  processes  followed 
during  this  phase,  the  phase  has  two  main  outputs: 

•  Concept  of  Operations  (ConOps).  The  ConOps  is  intended  to  capture  a  relatively  detailed 
operational  vision  of  how  a  capability  will  be  employed,  including  system  operation,  force 
generation  and  training,  maintenance,  command  and  control,  and  operational  structure,  to  a 
sufficient  level  of  detail  to  guide  further  development;  some  sample  text  from  the  JUSTAS 
ConOps  is  shown  in  Figure  4,  below.  It  is  the  authors’  opinion  that  ConOps  are  not  typically 
developed  as  requirements,  but  rather  as  a  detailed  account  of  the  envisioned  system 
sufficient  to  inform  the  development  of  operational  requirements. 


3  The  development  of  an  investment  plan  to  accompany  the  SCR  is  an  important  activity,  but  details  of 
investment  are  largely  outside  of  the  scope  of  this  research.  Interested  readers  are  referred  to  the  SCR 
document,  especially  Chapter  7,  for  details  of  how  the  SCR  is  transformed  into  an  investment  plan. 

4  In  military  parlance,  operators  are  to  be  distinguished  from  engineers.  Operators  are  the  personnel  who 
use  equipment  to  generate  effects  (e.g.,  flying  an  aircraft  to  deliver  munitions;  sailing  a  ship  to  project 
force;  using  an  infrared  sensor  to  gather  intelligence)  and  engineers  are  the  personnel  who  manage  that 
equipment  so  that  it  can  be  operated  (e.g.,  engine-room  artificers  on  a  ship;  aerospace  engineers  in  a 
squadron). 
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•  Statement  of  Operational  Requirement  (SOR).  The  SOR  follows  from  the  ConOps  and 
defines  the  operational  requirements  for  a  new  system  in  sufficient  detail  for  the  project  to 
move  from  the  development  of  operational  requirements  to  the  development  of  technical 
requirements.  The  SOR  should  be  clearly  linked  to  a  capability  deficiency  (see  Figure  5, 
below).  The  SOR  contains  a  large  number  of  detailed  operational  requirements  related  to  all 
aspects  of  system  operation,  maintenance,  training,  and  force  generation,  separated  into 
mandatory  and  rated  requirements5  (see  Figure  6,  below).  All  mandatory  requirements 
typically  trace  back  to  a  set  of  High-Level  Mandatory  Capabilities  (HLMCs;  see  Figure  7, 
below). 

While  these  documents  are  intended  to  fully  capture  the  operators’  understanding  of  the 
operational  requirements  for  the  system  to  be  procured,  there  are  four  important  considerations 
that  make  the  development  of  complete  and  consistent  operational  requirements  challenging. 

•  Lack  of  technical  context.  Operational  requirements  are  developed  by  personnel  who  do 
not  necessarily  have  a  good  understanding  of  the  technical  constraints  on  system  operation 
(and,  who  cannot  be  expected  to  have  that  knowledge).  This  can  lead  to  operational 
requirements  that  are  challenging,  or  impossible,  to  implement  from  a  technical  perspective. 

•  Impact  of  tacit  knowledge.  Operational  requirements  are  developed  by  personnel  who  have 
much  tacit  knowledge  about  the  area  for  which  they  are  developing  requirements.  This  tacit 
knowledge  may  never  be  properly  expressed  for  transfer  to  personnel  without  it. 

•  Narrow  inquiry.  Operational  requirements  are  developed  by  personnel  who  may  not 
understand  the  full  scope  of  concerns  to  account  for  in  requirements  development.  An 
operator  with  a  strong  interest  in  one  area  (for  example,  littoral  operations)  may  not  fully 
appreciate  the  concerns  of  other  areas  (for  example,  overland  operations).  As  a  result,  some 
requirements  categories  may  be  over-represented  while  others  are  under-represented. 

•  Task-artefact  cycle  effects.  The  operational  requirements  that  are  developed  are  frequently 
based  on  operators’  understanding  of  how  to  achieve  missions  with  similar  legacy 
equipment,  and  so  they  are  especially  prone  to  getting  caught  in  the  task-artefact  cycle  (see 
Section  1.1). 

To  help  mitigate  the  effects  of  these  challenges,  the  interface  between  the  development  of 
operational  and  technical  requirements  should  be  robust.  Not  only  should  the  relevant  documents 
be  passed  down  from  one  phase  to  the  next,  but  operational  personnel  familiar  with  the 
operational  requirements  for  a  new  system  should  be  employed  in  the  Project  Management  Office 
(PMO)  responsible  for  technical  requirements  development.  Unfortunately,  operational  personnel 
are,  in  practice,  rarely  employed  in  the  PMO,  making  this  interface  an  important  overall  project 
risk. 


5  Requirements  that  must  be  fulfilled  for  the  system  to  address  the  capability  deficiency  are  categorized  as 
mandatory,  whereas  requirements  that  do  not  bear  on  whether  or  not  a  system  meets  the  capability 
deficiency  but  rather  which  have  the  potential  to  improve  the  way  in  which  the  system  provides  its 
capability  are  categorized  as  rated.  In  later  stages  of  procurement,  mandatory  requirements  correspond  to 
items  that  a  potential  vendor  must  deliver  for  their  system  to  be  considered  compliant  to  the  Crown’s  needs, 
whereas  rated  requirements  are  scored  to  help  in  identifying  the  system  that  is  best  from  a  technical 
perspective. 


20 


DRDC  Toronto  CR  2010-049 


Esterline5  fCMC  Electronics 


3.3  Methods  of  UAV  Control.  Both  the  air  vehicle  and  the  oavload  are  rcmotelv 

controlled.  Control  is  normally  exercised  in  one  of  three  ways:  LOS,  BLOS,  or  autonomous. 

a. 

LOS  -  is  defined  as  the  condition  where  the  air  vehicle  (A  V)  is  controlled  via 
a  data  link  that  has  an  unobstructed  view  from  transmitter  to  receiver. 

Takeoffs  and  landings  are  pcrfomicd  using  LOS.  Mission  planning  for 
takeoffs  and  landings  is  critical  as  operating  altitude,  range  between  the 
receiver  and  transmitter,  surrounding  terrain  elevation,  ground  based 
transmitter  and  receiver  location  and  security  will  greatly  affect  the  UAV’s 
launch  and  recovery.  Missions  may  be  conducted  in  LOS  mode. 

b. 

BLOS  -  is  defined  as  the  condition  where  the  AV  and  pavload  are  controlled 
via  a  satellite  communications  data  link  and  /  or  other  extended 
communication  link  and  is  the  normal  control  mode  for  MALE  UAV 
operations.  The  UAV  mission  will  be  conducted  using  BLOS  communication. 

Two  new  factors  need  to  be  considered  with  this  control  method:  Satellite 

Visual  Horizon  (SVH  -  the  look  angle  from  the  UAV)  and  Satellite 

Transponder  availability. 

c. 

Autonomous  -  is  defined  as  Dre-oroaramnied  fliaht  profile  used  when  the 

UAV  self-manoeuvres  along  a  flight  profile.  It  could  be  employed  when  the 
command  link  is  lost  and  the  UAV  attempts  to  regain  the  command  link. 

Figure  4:  Some  sample  content  from  the  JUSTAS  ConOps. 

1.5  Deficiency 


1.5.1  Canadian  international  operations  have  highlighted  deficiencies  in  the 
provision  of  accurate  and  timely  intelligence,  surveillance,  reconnaissance  and 
target  acquisition  information  to  CF  commanders.  In  particular,  there  are 
insufficient  numbers  and  types  of  sensors  for  both  force  employment  and  force 
generation.  In  addition  to  sensor  deficiencies,  the  CF  does  not  have  an  extended 
endurance  precision  strike  capability  that  is  able  to  respond  to  time-critical 
emerging  targets.  These  deficiencies  have  the  potential  to  lead  to  significant 
gaps  in  the  understanding  of  the  operational  situation,  thereby  placing  mission 
success  and  CF  personnel  at  higher  risk.  The  acquisition  of  the  CU161  Sperwer 
tactical  UAV  (TUAV)  in  support  of  Operation  ATHENA  and  Operation  ARCHER 
provided  an  opportunity  to  assess  the  effectiveness  of  this  type  of  UAV  in 
addressing  some  of  these  deficiencies.  Sperwer  has  demonstrated  the 
effectiveness  of  a  UAV  in  a  hostile  environment;  however,  it  is  expected  to  be 
phased  out  by  Feb  2009  as  it  reaches  its  estimated  life  expectancy  (ELE). 

Project  Noctua  will  lease  the  CU170  Heron  as  an  interim  UAV  system  to  support 
Canadian  troops  in  Afghanistan.  The  Noctua  contract  will  be  for  two  years  of 
deployed  operations  with  options  to  extend  for  up  to  12  months  (Dec  2011).  To 
ensure  the  CF  does  not  experience  an  operational  gap  in  UAV  capability,  the 
acquisition  of  a  follow-on,  long-term  UAV  capability  to  address  deficiencies  is  a 
high  priority. 


Figure  5:  The  operational  deficiency  stated  in  the  JUSTAS  SOR. 
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5  SUB-SYSTEM  EFFECTIVENESS  REQUIREMENTS 

5.1  Air  Vehicle 

5.1.1  Mandatory  requirements: 

a.  The  air  vehicle,  at  reconnaissance  mission  weight  and 
configuration,  shall  be  capable  of  conducting  persistent  surveillance 
(a  minimum  12  hrs)  at  a  minimum  operating  radius  of  1000  km  from 
the  point  of  departure  and  return  with  sufficient  fuel  for  two  hours  of 
loiter  (zero  wind,  standard  atmosphere); 

b.  The  air  vehicle,  at  reconnaissance  mission  weight  and 
configuration,  shall  be  capable  of  transiting  1000  km  within  three  (3) 
hours  (zero  wind,  standard  atmosphere); 

c.  The  air  vehicle,  at  reconnaissance  mission  weight  and 


5.1.2  Rated  requirements.  The  UAV  system  will  be  assessed  with  respect  to  the 
following  air  vehicle  requirements: 

a.  Ability  of  the  air  vehicle,  at  reconnaissance  mission  weight  and 
configuration,  to  take  off  or  land  in  significantly  less  than  8000  ft; 

b.  Ability  of  the  air  vehicle,  at  reconnaissance  mission  weight  and 
configuration,  to  conduct  persistent  surveillance  in  excess  of  12 
hours  at  a  minimum  operating  radius  of  1000  km  from  the  point  of 
departure  and  return  with  sufficient  fuel  for  two  hours  of  loiter; 

c.  Ability  of  the  air  vehicle  to  operate  at  flight  altitudes  of  greater  than 
30,000  ft  (MSL)  at  mission  weight  and  configuration; 


Figure  6:  Examples  of  mandatory  and  rated  requirements  from  the  JUSTAS  SOR. 
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High  Level  Mandatory  Capabilities 

Surveillance  and  Target  Acquisition:  A  suite  of  sensors  that  enables  the  operator 
to  covertly  detect  and  identify  targets  and  obtain  targeting  data,  in  specified  adverse 
weather  conditions,  from  medium  altitude,  day  and  night. 


Range/Endurance:  Flying  endurance,  at  mission  weight,  to  ensure  a  relevant  radius 
of  operation  (a  minimum  of  1000  km)  and  allow  prolonged  (a  minimum  of  12  hours) 
on  station  surveillance. 


Operations  Tempo:  Sufficient  operational  and  training  systems,  personnel, 
infrastructure,  and  logistical  support  to  sustain  two  lines  of  tasking  at  a  single 
deployment  location,  as  well  as  force  generation  in  Canada. 


Interoperability:  Ability  of  the  system  to  provide  services  and  data  to,  and  accept 
services  and  data  from,  Joint  and  Combined  forces. 


Operational  Suitability:  Ability  to  conduct  sustained  operations  worldwide  in 
appropriate  classes  of  airspace,  under  specified  adverse  weather  conditions,  and  in 
low-to-medium  threat  environments.  Provides  spectrum  supportability  in  Canada  and 
abroad. 


Dynamic  Control  and  Responsiveness:  Ability  to  dynamically  control  the  air 
vehicle  and  payload,  in  near-real  time,  under  line  of  sight  (LOS),  beyond  line  of  sight 
(BLOS),  and  remote  split  operations,  and  respond  rapidly  to  situational  changes  and 
new  taskings. 


Flexibility/Growth  Capacity:  The  air  vehicle  must  possess  the  demonstrated 
flexibility,  growth  capacity,  and  standard  interfaces  required  to  integrate  new 
payloads  to  support  enhanced  overland  capabilities  and  maritime  domain  awareness. 


Force  Application:  Capable  of  enabling  Joint  Fires,  and  carrying  and  employing 
precision-guided  munitions  (PGM). 


Figure  7:  The  HLMCs  from  the  JUSTAS  SOR. 


2.2.6  Technical  requirements  development 

Once  the  operational  requirements  for  a  new  system  have  been  developed,  the  procurement  effort 
goes  through  a  penultimate  round  of  funding  decisions  to  determine  if  the  project  should  proceed 
to  the  level  of  technical  requirements  development.  If  a  project  gets  to  this  stage,  it  indicates  a 
high  level  of  commitment  by  the  CF  (and  the  Treasury  Board)  to  the  project,  as  the  outputs  of  this 
phase  are  the  technical  aspects  of  the  Request  for  Proposal  (RFP)  that  will  be  submitted  to 
vendors  for  the  purpose  of  eliciting  bids,  and  ultimately,  selecting  a  contractor  to  provide  the 
system. 

Technical  requirements  development  is  supported  by  a  PMO,  whose  responsibility  is  to 
decompose  the  operational  requirements  into  technical  requirements  that  specify  the  attributes  of 
the  system  to  a  level  of  detail  that  supports  procurement.  This  is  a  challenging  task.  Looking  up 
the  requirements  chain  to  the  operational  requirements,  the  engineers  developing  technical 
requirements  must  have  a  strong  grasp  of  the  operational  requirements.  As  mentioned  in  Section 
2.2.5,  this  is  supported  by  supplementing  the  PMO  with  operational  personnel.  However,  even 
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with  this  support  it  can  be  expected  that  there  will  be  challenges  in  bridging  from  operational  to 
technical  requirements  because  operators  and  engineers  come  at  the  work  from  different 
perspectives.  Looking  down  the  requirements  chain  to  the  implementation  phase,  it  can  be 
challenging  to  develop  appropriately  detailed,  complete,  and  consistent  technical  requirements. 
Requirements  specified  at  a  high-level  may  have  a  higher  likelihood  of  providing  complete 
coverage  of  the  problem  space,  but  they  may  provide  vendors  with  so  much  flexibility  that  the 
resultant  system  will  not  meet  the  CF’s  operational  needs.  Conversely,  requirements  specified  at  a 
low-level  may  have  a  higher  likelihood  of  satisfying  the  CF’s  operational  needs,  but  they  may 
also  constrain  vendors  so  severely  that  the  system  may  end  up  costing  significantly  more  than  if 
vendors  were  allowed  more  flexibility. 

Two  types  of  documents  typically  result  from  this  phase.  First  is  a  project  SOW,  that  defines  the 
how  the  work  to  deliver  the  system  should  be  undertaken  by  a  vendor.  Second  are  the  technical 
specifications  that  define  what  system  should  be  delivered.  The  package  of  documentation  that 
results  from  this  phase  of  work  can  be  extensive  and  detailed.  For  example,  the  Maritime 
Helicopter  (MH)  RFP  (released  on  1 1  April  2003)  included  13  volumes,  as  follows: 

•  Volume  1  -  General  instructions  to  bidders 

•  Volume  2  -  Proposal  requirements 

•  Volume  3  -  Evaluation  plan 

•  Volume  4  -  MH  model  contract 

•  Volume  5  -  MH  SOW 

•  Volume  6  -  MH  Requirements  Specification  (MHRS) 

•  Volume  7  -  MH  statement  of  operating  intent 

•  Volume  8  -  MH  Contractor  Data  Requirements  List  (CDRL)  and  Data  Item  Descriptions 
(DIDs) 

•  Volume  9  -  MH  in-service  support  model  contract 

•  Volume  10  -  MH  in-service  support  SOW 

•  Volume  1 1  -  Life -cycle  support  requirements  specification 

•  Volume  12  -  Statement  of  support  intent 

•  Volume  13  -  In-service  support  CDRL  and  DIDs 

While  only  one  of  the  volumes,  the  MHRS  (Volume  6)  focuses  on  technical  requirements,  this 
volume  has  256  pages,  over  113,000  words,  and  14  appendices  (some  sample  requirements  from 
this  document  are  provided  in  Ligure  8,  below).  Within  a  set  of  technical  requirements  of  this 
breadth,  implementation  efforts  will  certainly  reveal  problems  of  inconsistency  and 
incompleteness.  Even  though  the  staff  working  on  these  requirements  were  likely  diligent  and 
motivated,  requirements  engineering  is  a  wicked  problem6  (Bubenko,  1995;  Yeh,  1984)  for  which 
even  the  definition  of  what  constitutes  a  correct  solution  does  not  exist. 


6  With  respect  to  requirements  engineering  for  software  projects,  Yeh  (1984)  characterizes  a  problem  as 
wicked  when  it  has  one  or  more  of  the  following  five  characteristics:  (1)  the  problem  formulation  and  its 
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3.10.3  Automatic  Transition 

3.10.3.1  AFCS  automatic  transition  to  a  pilot-selected  ground  speed,  heading  and  either  radar  altitude 
or  barometric  altitude  shall  be  provided  from  anywhere  inside  the  SERVICE  ENVELOPE. 

3.10.3.2  The  AFCS  automatic  transition  shall  generate  a  maximum  rate  of  descent  of  500  feet  per 
minute  during  any  transition  phase. 

3.10.3.3  The  MH  shall  be  able  to  perform  an  AFCS  automatic  transition  to  a  helicopter  generated  "into 
wind"  hover. 

3.10.3.4  The  automatic  transition  shall  take  less  than  95  seconds  from  a  gate  of  150  feet  AGL  at  120 
KIAS  down  to  the  selected  and  programmed  hover  altitude. 

3.10.3.5  The  automatic  transition  "into  wind"  heading  shall  be  achieved  prior  to  the  helicopter 
descending  below  100  feet  AGL. 

3.10.3.6  The  MH  shall  be  able  to  conduct  an  AFCS  automatic  transition  within  five  minutes  after  first 
launch  from  a  HFX  in  conditions  described  in  Section  3.5.1  of  this  RS. 


Figure  8:  A  subset  of  the  requirements  for  the  Automatic  Flight  Control  System  (AFCS)  from 

the  MHRS. 


2.2.7  Implementation 

The  phases  of  procurement  described  so  far  can  all  be  considered  to  be  within  the  problem-space 
of  procurement,  and  define,  with  increasing  levels  of  specificity,  the  procurement  problem  the 
Crown  needs  to  resolve.  The  question  of  interest  is,  “How  can  the  Crown’s  needs  best  be 
described?”  As  soon  as  an  RFP  for  a  procurement  project  is  released  to  the  vendor  community, 
the  work  of  requirements  engineering  transitions  from  the  problem-space  to  the  solution-space. 
The  important  question  here  is,  “What  system  can  industry  propose  and  provide  that  best  meets 
the  Crown’s  needs?” 

Working  specifically  from  the  requirements  perspective,  the  vendor  ultimately  selected  to  provide 
the  Crown  with  a  new  system  uses  the  Crown’s  requirements  as  a  starting  point  to  develop  a 
design  that  satisfies  the  Crown’s  requirements.  Assuming  a  project  that  involves  hardware  and 
software  components,  vendors  will  generate  their  own  set  of  high-level  requirements  with  an 
organization  and  terminology  that  conforms  to  their  chosen  system  architecture  and  that  also 
responds  to  all  of  the  Crown’s  requirements.  Following  from  this,  requirements  will  be  developed 
to  a  very  low  level  to  actually  specify  the  design  that  the  vendor  will  provide.  For  example,  it  is  a 
best  practice  that  one  low-level  software  requirement  be  written  to  drive  the  production  of  every 
approximately  10  lines  of  code. 

As  requirements  are  developed  to  increasingly  low-levels  of  design  detail,  vendors  typically 
encounter  at  least  the  following  three  classes  of  problems  in  understanding  and  implementing  the 
Crown’s  requirements: 


solution  cannot  be  separated;  (2)  there  are  no  rules  to  determine  when  a  solution  is  complete;  (3)  symptoms 
and  causes  cannot  be  distinguished;  (4)  the  problem  is  substantially  unique;  and,  (5)  exhaustive  and  definite 
problem  formulation  are  not  possible.  This  leads  Yeh  to  observe,  “These  five  characteristics  lead  to  a  set  of 
nasty  attributes  which  are  common  among  large  [technology]  systems.  Such  systems  are:  hard  to  define, 
hard  to  plan,  hard  to  manage,  hard  to  schedule,  [and]  hard  to  test.  (p.  18).” 
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•  Interpretation.  The  Crown’s  requirements  may  have  seemed  clear  at  the  time  they  were 
written.  However,  when  those  same  requirements  are  read  by  the  vendor,  whose  personnel 
have  different  backgrounds  and  experiences,  they  may  be  difficult  to  interpret. 

•  Inconsistency.  The  Crown’s  requirements  may  also  have  seemed  consistent  at  the  time  they 
were  written.  However,  when  those  same  requirements  are  viewed  through  the  lens  of  the 
vendor’s  proposed  solution,  some  of  them  may  seem  inconsistent. 

•  Incompleteness.  Finally,  it  is  difficult  to  imagine  that  a  PMO  would  issue  a  set  of 
requirements  that  they  felt  were  incomplete  in  any  substantial  way.  However,  as  a  design 
coalesces  around  a  set  of  requirements,  incomplete  requirements  may  be  exposed. 

In  the  authors’  opinion,  these  problems  have  three  root  causes: 

•  Tacit  knowledge.  CF  personnel  who  develop  requirements  typically  have  a  great  deal  of 
tacit  knowledge  about  the  problem  space  that  may  not  be  shared  by  vendor  personnel.  Since 
this  knowledge  is  tacit,  there  is  a  low  likelihood  that  CF  personnel  reviewing  a  requirements 
document  will  notice  that  it  is  missing. 

•  Unstated  assumptions.  Related  to  tacit  knowledge,  CF  personnel  may  also  have 
assumptions  about  how  they  would  expect  a  requirement  to  be  interpreted  or  a  problem  to  be 
solved.  While  some  of  these  assumptions  may  be  shared  by  vendor  personnel,  others  may 
not  be. 

•  The  gulf  between  the  problem  space  and  the  solution  space.  As  CF  personnel  develop  a 
set  of  requirements,  it  is  likely  that  each  person  involved  in  the  process  has  their  own  mental 
model  of  what  they  expect  the  solution  to  be,  and  define  their  portion  of  the  problem  around 
that  solution.  In  cases  where  these  mental  models  do  not  conform  with  the  chosen  solution, 
there  will  likely  be  a  mismatch  as  attempts  are  made  apply  the  requirements  defined  around 
one  solution  to  another  solution. 

These  problems  are  typically  solved  either  by  the  vendor  imposing  an  interpretation  and  taking  it 
on  risk  that  the  interpretation  was  correct  (which  is  often  a  reasonable  thing  to  do  to  ensure  that  a 
project  meets  its  cost  and  schedule  targets),  or  by  the  vendor  conferring  with  the  Crown  through  a 
memorandum  or  a  working-group  meeting.  Interestingly,  conferring  with  the  Crown  may  not  lead 
directly  to  a  correct  interpretation  of  a  requirement.  Due  to  the  cycle  of  military  postings,  the 
personnel  involved  in  implementation-level  working-group  meetings  will  likely  not  be  the  same 
personnel  involved  in  writing  the  technical  requirements. 

As  a  final  note,  most  major  procurement  projects  do  not  involve  just  a  single  vendor,  but  rather  a 
vendor  consortium  with  a  prime  contractor  and  many  sub-contractors.  While  this  arrangement  is 
typically  the  only  way  for  vendors  to  successfully  respond  to  a  major  Crown  RFP,  it  only 
exacerbates  the  problems  noted  in  this  section. 


26 


DRDC  Toronto  CR  2010-049 


EsterlineJ-  fCMC  Electronics 


2.3  Requirements  engineering  and  options  analysis 

2.3.1  Introduction 

Challenges  in  CF  procurement  may  overlap  with  those  encountered  in  other  domains  and 
discussed  in  the  academic  literature.  To  ensure  that  this  R&D  roadmap  incorporates  best  practices 
from  other  fields,  does  not  duplicate  prior  work,  and  addresses  questions  of  interest  to  the 
requirements  engineering  community,  we  briefly  review  the  requirements  engineering  and 
options  analysis  literature. 

First,  we  note  some  differences  between  the  domains  in  which  requirements  engineering  is 
practiced  and  the  CF  with  respect  to  workflow,  client-vendor  relations,  and  complexity  of 
projects.  Next,  we  provide  a  brief  survey  of  the  requirements  engineering  methods  advocated  by 
researchers  and  those  used  in  practice.  Finally,  we  discuss  work  already  completed  in  applying 
CWA  methods  in  military  procurement,  and  opportunities  to  build  on  this  progress  in  the  R&D 
roadmap. 

2.3.2  Domains  &  comparison  to  CF 

Requirements  engineering  is  not  a  stand-alone  profession,  and  at  least  three  main  communities  of 
practice  contribute  to  the  literature:  software  development,  business  project  management,  and 
sociotechnical  systems  practice.  Before  discussing  requirements  engineering  and  options  analysis 
methods,  it  is  useful  to  consider  some  of  the  work  demands  that  have  influenced  their  evolution. 

Software  development  has  a  rich  history  of  requirements  engineering  research  field  (Cheng  & 
Atlee,  2007),  in  part  due  to  the  complexity  of  software  projects,  and  in  part  because  of  the 
difficulty  of  efficiently  and  completely  implementing  requirements  in  machine  code.  The  need  for 
efficiency  is  self -reinforcing  in  two  ways:  companies  using  efficient  requirements  engineering 
methods  may  be  more  profitable,  and  widely-adopted  requirements  engineering  methods  will  be 
more  likely  to  attract  efficiency-boosting  support  tools.  Unlike  requirements  engineering  methods 
in  the  CF,  software  methods  deal  primarily  with  low-level  implementation  requirements. 

Business  project  management  requirements  engineering  methods  are  discussed  more  by 
industry  groups  (Haskins,  2007)  and  management  consultants  (Gilb  &  Gilb,  2007;  Halligan, 
undated;  Philbin,  2008)  than  in  the  academic  literature.  Much  business  requirements  engineering 
is  concerned  with  managing  clients’  expectations,  smoothing  contractual  negotiations,  and 
supporting  cost  and  effort  estimation.  Some  organizations  like  the  National  Aeronautics  and 
Space  Administration  (NASA)  develop  first-of-a-kind  systems,  but  this  is  uncommon.  Business 
practices  encourage  early  and  continuous  client-vendor  communication  to  elicit  and  negotiate 
requirements. 

Finally,  socio-technical  systems  and  cognitive  engineering  practice  is  often  discussed  from  a 
consultant’s  or  internal  champion’s  perspective.  More  has  been  written  about  how  requirements 
should  be  realized  (e.g.,  the  literature  on  developing  effective  designs  (Crandall,  Klein,  & 
Hoffman,  2006)  and  clearly  communicating  design  specifications  (Miller  &  Vicente,  2001))  than 
about  developing  the  requirements  themselves.  Even  though  this  is  the  case,  much  of  this  work 
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has  been  done  and  has  been  effective  within  military  contexts  (e.g.,  Bisantz  et  al.,  2003; 
Chalmers,  Easter,  &  Potter,  2000;  Hagenaars,  2003;  Miller,  2004). 

2.3.3  Requirements  engineering  and  options  analysis  techniques  used 

The  fields  of  practice  described  above  have  cultivated  different  requirements  engineering  and 
options  analysis  methods.  We  considered  these  methods  as  functional,  goal-oriented,  expert- 
mediating,  or  work-analysis  based.  Each  reflects  the  needs  of  a  particular  work  practice,  and 
suggests  implications  for  research  and  development  of  requirements  engineering  and  options 
analysis  methods. 

•  Functional  requirements  engineering.  Functional  requirements  engineering  approaches 
are  most  often  discussed  in  the  software  engineering  community,  and  include  modeling  tools 
such  as  the  Unified  Modeling  Language  (UML)  (van  Lamsweerde,  2000)  and  Systems 
Modeling  Language  (SysML)  (Haskins,  2007).  They  are  typically  applied  by  software 
developers  through: 

♦  Developing  use  cases,  each  a  set  of  goal-directed  scenarios. 

♦  For  each  scenario,  identifying  domain  concepts  &  relationships  between  concepts 
required  to  accomplish  the  scenario  goal. 

♦  Translating  the  set  of  concepts  and  relationships  into  system  functional  requirements 
to  be  implemented  with  compatible  methods  such  as  object-oriented  programming. 

Functional  requirements  engineering  approaches  seek  to  describe  only  what  a  system  is 
required  to  do,  using  rigorous  structures  that  can  be  manipulated  by  support  tools  to  manage 
large  requirement  sets  (INCOSE  Tools  Database  Working  Group,  undated),  and  translated 
into  software  implementations  with  minimal  re-work  (Castro,  Kolp,  &  Mylopoulos,  2002). 
However,  this  well-formed  structure  is  achieved  at  the  cost  of  omitting  requirements  that 
cannot  be  described  in  terms  of  functions,  such  as  security,  safety,  or  fault -tolerance.  Such 
non-functional  requirements  are  omitted  from  modeling  (Castro,  et  al.,  2002)  and  relegated 
to  supporting  documents.  Also  omitted  are  the  whys  underlying  requirements,  which  may 
hamper  third  parties  in  charitably  interpreting  requirements  and  evaluating  design  options. 

•  Goal-oriented  requirements  engineering.  Deficiencies  in  functional  requirements 
engineering  methods  have  been  described  at  length  to  justify  development  of  goal-directed 
software  requirements  engineering  methods.  Two  examples  are  KAOS,  a  goal-augmented 
functional  method  (van  Lamsweerde,  2000)  and  i*,  a  social  dependency-oriented  method 
(Yu,  1997).  Both  include  analysis  frameworks  and  graphical  notations  for  describing  both 
functional  and  non-functional  system  requirements  and  the  goal-directed  relations  between 
them.  However,  neither  has  been  adopted  in  practice,  and  we  could  not  find  application 
examples  for  high-level  sociotechnical  system  requirements. 

The  history  of  software  requirements  engineering  methods  suggests  two  implications  for  a 
requirements  engineering  R&D  roadmap.  Firstly,  methods  to  manage  high-level 
requirements  should  include  a  structured  representation  of  the  whys  that  justify  functional 
requirements.  Secondly,  any  requirements  engineering  method  will  be  more  easily  adopted 
if  its  notations  and  intermediate  products  are  useful  to  others  besides  requirements 
engineers. 
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•  Expert-mediating  methods.  The  intentions  behind  functional  requirements  can  often  be 
inferred  by  domain  experts,  and  one  category  of  requirements  engineering  and  options 
analysis  methods  leverages  this.  Such  expert-scorekeeping  methods  help  mediate  and 
organize  domain  experts’  evaluations  using  simple  function-priority  or  risk-cost  taxonomies 
and  spreadsheet-like  propagation  of  experts’  cost  or  probability  estimates.  Similar  methods 
are  used  to  draft  early-stage  requirements  for  first-of-a-kind  NASA  missions  (Feather  & 
Cornford,  2003)  and  to  manage  contract  work  by  producers  of  consumer  equipment  (Gilb  & 
Gilb,  2007;  Philbin,  2008). 

These  methods  are  intended  to  be  used  early  in  the  requirements  process,  and  can  blur  the 
distinction  between  options  analysis  and  requirements  engineering.  As  long  as  a 
comprehensive  expert  team  can  be  assembled,  abstract  and  ill-defined  requirements  can  be 
reconciled  without  support  from  theory-guided  analyses.  If  expert  assessment  is  not 
effective  or  requirements  are  contractually  sensitive,  these  methods  suggest  specifying 
requirements  in  terms  of  empirical  user  performance  criteria  (Gilb  &  Gilb,  2007). 

Three  implications  from  this  field  of  practice  are  that: 

•  options  analysis  may  occur  in  parallel  with,  and  is  not  typically  referred  to  as  separate  from, 
requirements  engineering; 

•  expert  judgment  can  incorporate  intent  and  abstract  requirements,  but  existing  representation 
tools  may  not  effectively  communicate  their  deliberations;  and, 

•  if  human-centered  requirements  cannot  be  empirically  tested,  such  as  in  early-stage  CF 
procurement,  simulation  methods  may  hold  promise. 

The  final  field  of  practice  that  we  reviewed  was  sociotechnical  systems  design  and  cognitive 
engineering.  Discussion  of  requirements  in  these  fields  has  typically  been  taken  to  mean 
computer  interface  ‘information  requirements’  (Miller  &  Vicente,  2001),  or  has  been  conflated 
with  design  specifications.  Methods  in  the  sociotechnical  systems  design  and  cognitive 
engineering  fields  tend  towards  supporting  creative  designer  judgment  (Crandall,  et  al.,  2006)  or 
specifying  analysis  steps  intended  to  ensure  effective  designs  (Gualtieri,  Szymczak,  &  Elm, 
2005). 

An  implication  from  this  practice  has  already  been  recognized  by  the  software  and  product  design 
practice:  requirements  (what)  must  be  distinguished  from  design  specifications  (how)  (Gilb  & 
Gilb,  2007). 

2.3.4  Case  study  of  CWA  for  requirements  engineering 

In  Section  2.4,  below,  we  discuss  the  application  of  CWA,  a  set  of  analysis  techniques  and 
modelling  tools  for  analysing  complex  sociotechnical  systems,  to  requirements  engineering.  In 
our  review  of  the  requirements  engineering  literature  we  learned  of  a  previous  successful 
application  of  CWA  to  requirements  engineering  and  options  analysis  by  the  Australian  Defense 
Science  and  Technology  Organization.  Published  accounts  include  tender  evaluation  (Naikar  & 
Sanderson,  2001)  and  crew  configuration  (Naikar,  Pearce,  Drumm,  &  Sanderson,  2003),  while 
work  is  ongoing  on  evaluating  UAV  staffing  requirements  (Elix  &  Naikar,  2009).  Details  of 
methods  are  described  in  Annex  A,  so  only  a  brief  overview  is  discussed  here. 
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The  first  application  used  Work-Domain  Analysis  (WDA)  as  an  options  analysis  support  tool. 
Similar  to  expert-scorekeeping  methods,  technical  focus  groups  first  assessed  the  subsystem-level 
functionality  of  each  option.  The  resulting  evaluations  were  then  abstracted  using  WDA  means- 
ends  links  to  help  estimate  the  effect  of  each  subsystem’s  functionality  on  system  processes, 
values,  and  purposes.  Next,  WDA  was  used  to  structure  a  comparison  between  all  options  for 
evaluation  by  a  military  decision-making  committee  (Naikar  &  Sanderson,  2001).  The  structured 
translation  from  concrete,  specialized  functions  to  abstract,  operationally  relevant  processes  and 
values  helped  manage  a  project  of  large  scope  without  software  support  tools,  and  also  helped  to 
coordinate  small  teams  of  specialized  technical  experts.  While  this  work  was  successful,  it  has 
some  important  limitations  in  the  context  of  the  current  work:  it  considered  only  options  analysis, 
and  not  requirements  generation;  it  required  high-level  commitment  of  a  kind  that  may  be 
difficult  to  get  within  the  CF  for  this  type  of  work;  it  was  very  time-consuming;  and  no  follow¬ 
ups  or  duplications  have  been  reported  in  the  literature.  Further,  it  is  possible  that  a  judicious 
application  of  M&S  could  have  packaged  the  insights  of  experts  and  so  increased  overall 
efficiency  by  reducing  the  amount  of  time  that  experts  were  required  to  spend  on  this  project. 

A  second  application  used  Control  Tasks  Analysis  (ConTA),  WDA,  and  Social  Organization  and 
Cooperation  Analysis  (SOCA)  to  evaluate  crewing  requirements.  First,  scenarios  were  developed 
to  put  work  situations  and  activities  in  context,  similar  to  a  functional  requirements  engineering 
approach.  However,  ConTA  was  used  to  systematically  develop  as  comprehensive  scenarios  as 
possible.  Experts  then  assessed  how  each  crewing  configuration  would  affect  scenario  response, 
and  estimated  effects  on  WDA-derived  criteria  (Naikar,  et  al.,  2003).  ConTA  was  then  used  to 
systematically  integrate  scenario-specific  evaluations  into  more  general  activities  and  situations. 
By  comparing  crew  configurations  on  measures  such  as  frequency  of  role  reallocation  and 
quantity  of  overhead  communication,  the  most  promising  social  organization  was  justified  to 
operational  staff  and  included  in  the  vendor’s  system  design.  Again,  while  this  work  was 
successful,  it  suffers  from  some  limitations  in  the  context  of  Project  14dj:  the  work  was  done  in 
the  implementation  phase  and  not  during  requirements  development,  and  it  again  required 
significant  staff  commitments  to  organize  and  participate  in  multiple  tabletop  focus  group 
sessions. 

This  R&D  roadmap  should  build  on  the  demonstrated  success  of  CWA  in  a  military  acquisition 
context.  Opportunities  for  future  work  identified  by  software  practitioners  (Cheng  &  Atlee,  2007) 
and  justifications  given  for  using  CWA  overlap  significantly,  suggesting  that  CWA  applications 
will  address  research  questions  of  interest  to  many  practitioners. 

First,  since  human  &  technology  requirements  are  interdependent  and  interacting,  requirements 
engineering  frameworks  must  be  developed  to  capture  both  intentions  and  technology  (Cheng  & 
Atlee,  2007;  Naikar  &  Sanderson,  2001).  However,  Naikar’s  work  was  completely  independent 
from  any  simulation  evaluations  of  technological  requirements,  and  only  the  final  evaluation 
committee  integrated  the  findings  (Naikar  &  Sanderson,  2001).  Extending  CWA  to  better 
integrate  M&S  capabilities  might  improve  the  usefulness  of  M&S  results  and  reduce  the  time 
required  for  their  application. 

A  second  justification  for  CWA  in  requirements  engineering  is  that  military  systems  require 
flexible,  innovative  problem  solving  and  are  more  likely  to  be  first-of-a-kind  systems.  Thus 
requirements  engineering  methods  that  are  scenario  or  task-dependent  may  not  capture 
functionality  needed  for  flexibility  (Naikar  &  Sanderson,  2001),  and  expertise  that  might 
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anticipate  such  deficiencies  may  not  yet  exist  (Naikar,  et  al.,  2003).  Software  requirements 
engineering  practitioners  (Cheng  &  Atlee,  2007)  also  recognize  the  need  to  specify  requirements 
for  adaptability  and  fault-tolerance.  However,  CWA  methods  to  date  still  rely  heavily  on  priming 
experts  with  scenarios,  and  comprehensive  scenario  development  may  not  be  possible  for  larger 
scope  projects.  Methods  for  narrowing  the  scope  of  scenario-primed  expert  judgment  may  help  in 
making  more  effective  use  of  scare  domain  expert  resources  and  improve  requirements 
engineering  effectiveness.  This  could  be  accomplished  through  more  effective  interview  notations 
(Elix  &  Naikar,  2009;  Naikar,  Moylan,  &  Pearce,  2006),  by  using  modeling  capabilities  to 
construct  a  minimal  set  of  critical  scenarios,  or  by  using  simulation  capabilities  to  animate 
scenario  operations,  workload,  and  crewing  allocations  to  prime  expert  evaluations. 

2.3.5  Insights  from  the  literature 

The  following  is  a  list  of  some  of  the  insights  from  the  requirements  engineering  and  options 
analysis  literature  that  could  be  relevant  in  the  context  of  Project  14dj: 

•  Options  analysis  is  best  considered  as  another  type  of  requirements  engineering.  While 
there  may  be  times  when  the  requirements  engineering  process  is  more  or  less  in  the  mode 
of  trading  off  options  against  one  another,  the  fact  that  a  requirement  is  needed  implies  that 
there  are  some  other  options  that  are  being  rejected.  This  implies  that  any  techniques  that 
assist  in  requirements  engineering  have  promise  to  assist  options  analysis,  and  vice-versa. 

•  Requirements  need  context.  Current  requirements  engineering  practice  seems  to  be  to 
ensure  that  requirements  are  measurable  statements  of  what  should  be,  while  omitting  why 
the  what  should  be  and  how  the  what  should  happen.  While  this  supports  the  development  of 
testable  requirements,  it  also  increases  the  potential  that  requirements  will  be 
misunderstood. 

•  Requirements  do  not  always  benefit  from  context.  While  requirements  should  be 
presented  along  with  their  context,  requirements  notation  should  separate  intent  (why)  and 
specifications  (how)  from  requirements  (what).  This  will  ensure  that  aspects  of  the 
requirements  that  need  to  be  testable  and  measurable  stay  that  way. 

•  Requirements  have  a  larger  audience  than  just  HF  practitioners.  For  a  requirements 
engineering  method  to  be  adopted,  relevance  to  product  development  workflow, 
minimization  of  contractual  risks,  and  efficiency  of  use  are  important  criteria. 

•  M&S  can  assist  in  the  development  of  measurable  and  testable  human-related 
requirements.  Since  M&S  techniques  are  inherently  focused  on  the  use  of  metrics  to 
evaluate  performance,  M&S  efforts  can  assist  in  developing  measurable  and  testable 
requirements  by  contributing  and  providing  rationale  for  performance  measures. 

2.4  Cognitive  Work  Analysis 

2.4.1  Introduction 

CWA  is  a  set  of  analysis  techniques  and  modelling  tools  for  analysing  complex  sociotechnical 
systems.  CWA  is  a  formative  framework,  which  means  that  it  is  especially  useful  for  helping  to 
form  an  understanding  of  a  how  an  envisioned  system  could  work.  While  it  is  typically  positioned 
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in  opposition  to  other  forms  of  HF  analysis  (such  as  MIL-HDBK-46855  Mission,  Function,  and 
Task  analysis),  CWA  is  best  thought  of  as  complementary  to  these  techniques.  CWA’s  unique 
contribution  to  the  HF  toolset  is  a  focus  on  the  essential  constraints  of  the  workspace:  it  describes 
not  what  people  should  do,  or  what  they  actually  do,  but  rather  tries  to  capture  the  constraint 
space  that  describes  all  that  people  could  do. 

The  purpose  of  this  section  is  not  to  present  a  detailed  account  of  CWA  (this  has  already  been 
done  elsewhere  in  the  literature;  see  Burns  &  Hajdukiewicz,  2004;  Lintern,  2009;  Naikar, 
Hopcroft,  &  Moylan,  2005;  Rasmussen,  et  al.,  1994;  Vicente,  1999),  but  rather  to  pose  some 
questions  on  the  way  in  which  CWA  could  be  applied  to  requirements  engineering  and  options 
analysis  for  CF  procurement.  We  generated  the  questions  listed  below  in  a  group  discussion  in 
which  we  worked  through  the  five  phases  of  CWA  to  develop  thoughts  on  how  each  phase  could 
potentially  be  applied  to  and  benefit  requirements  engineering  and  options  analysis  for  CF 
procurement.  The  following  two  sections  present  a  number  of  insights  about  CWA  that  could  be 
relevant  to  Project  14dj. 

2.4.2  Insights  about  CWA  in  the  context  of  CF  procurement 

The  following  is  a  list  of  some  of  the  insights  about  CWA  that  could  be  relevant  in  the  context  of 
Project  14dj: 

•  Importance  of  and  potential  for  functional  analysis.  Whether  it  is  implicit  or  explicit,  all 
requirements  development  must  be  based  on  a  functional  model  of  the  system  under 
analysis.  In  our  review  of  CF  procurement,  we  noted  that  requirements  development  may 
frequently  be  based  on  an  assumed  (implicit)  implementation  model,  but  we  did  not  find  any 
evidence  of  any  explicit  functional  modelling7  outside  of  the  current  proposal  to  use  the 
DNDAF.  We  also  noted  the  challenges  that  are  caused  by  the  tacit  knowledge  and  unstated 
assumptions  that  are  involved  in  each  phase  of  requirements  development.  In  this  context,  it 
is  possible  that  the  first  phase  of  CWA,  WDA,  could  be  used  to  develop  an  explicit 
functional  model  of  the  system  under  analysis.  Not  only  will  this  make  explicit  a  model  that 
must  already  be  developed  implicitly,  it  could  also  point  to  and  clarify  the  presence  of  tacit 
knowledge  or  assumptions  in  the  requirements  development  process.  Further,  because  WDA 
produces  models  that  are  relevant  to  understanding  human  decision-making,  this  effort 
would  help  to  insert  an  early  emphasis  on  human  performance  and  decision-making  into  the 
requirements  engineering  process.  Finally,  if  the  analysis  were  explicit,  there  is  a  greater 
likelihood  that  it  would  be  more  comprehensive  and  would  include  consideration  of  all 
relevant  factors. 

•  Relevance  of  why  and  how  to  understanding  what  a  requirement  is  about.  As  discussed 
in  the  review  of  requirements  engineering  and  options  analysis  (Section  2.3),  requirements 
engineering  practice  prefers  to  develop  requirements  that  capture  only  what  is  required, 
stripping  out  why  that  is  required  and  how  it  will  be  provided.  However,  it  is  likely  that 
stripping  this  context  from  requirements  may  make  it  more  difficult  for  participants  in  the 
process  to  know  where  the  requirements  were  affected  by  tacit  knowledge  and  unstated 
assumptions.  It  is  possible  that  structuring  requirements  around  the  Abstraction- 
Decomposition  Space  (ADS),  and  especially  by  focusing  on  the  means-ends  links  (that 

7  We  suspect  that  our  finding  is  more  indicative  of  the  fact  that  our  research  was  limited  than  that  no 
functional  modeling  is  a  part  of  the  CF  procurement  process.  More  research  is  needed  here. 
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provide  the  why  and  how  context  for  the  what  of  each  node  in  the  model)  may  help  to 
provide  this  important  context. 

•  Challenges  in  the  application  of  CWA.  Applying  CWA  is  challenging,  both  because  the 
modelling  representations  (e.g.,  the  ADS  and  the  Decision  Ladder)  have  important 
subtleties,  and  because  it  is  based  on  a  challenging  theoretical  framework.  If  CWA  is  to  be 
applied  to  military  procurement,  it  is  likely  that  tool  support  will  be  required  to  help  ensure 
that  the  process  of  modelling  is  understandable,  and  that  the  tool  support  may  need  to  be 
based  on  a  version  of  CWA  with  terminology  specifically  tailored  to  CF  procurement. 

•  The  function  of  blank  boxes  in  CWA  models.  In  our  discussions  about  the  application  of 
CWA  to  requirements  engineering,  we  suspected  that  ADS  models  and  Decision  Ladders 
done  in  the  requirements  phase  may  include  system  structures  or  operations  that  cannot  be 
consistently  or  completely  described.  For  example,  the  actual  physical  form  of  a  system  may 
not  need  to  be  prescribed  in  the  requirements  (which  will  lead  to  blank  boxes  in  the  physical 
form  level  of  an  ADS),  or  the  full  set  of  activators  to  a  decision-making  process  may  be 
unknown  or  only  possible  to  define  during  the  implementation  phase  (leading  to  an 
underspecified  Activation  box  in  the  Decision  Ladder).  It  is  possible  that  these  blank  boxes 
will  be  able  to  help  in  identifying  and  bounding  the  unknowns  of  a  procurement  project. 

•  CWA  and  scenarios.  Previous  work  (Torenvliet  &  Jamieson,  2007)  has  identified  the 
potential  for  the  ADS  to  support  the  development  of  scenarios  that  exercise  relevant  aspects 
of  the  work  domain.  While  this  work  was  performed  in  the  context  of  a  detailed  analysis  of 
crewing  levels  for  a  new  naval  platform,  it  is  possible  that  it  could  be  extended  to  support 
the  development  of  better,  more  comprehensive  scenarios  for  CBP  or  operational 
requirement  development. 

•  Potential  for  using  ConTA  as  a  point-of-departure  for  CWA.  Naikar,  Moylan,  and 
Pearce  (2006)  have  produced  helpful  guidance  on  the  application  of  ConTA  that  includes 
the  concept  of  work  situations  and  have  proposed  a  question-based  annotation  method  that 
can  be  more  consistently  interpreted  across  situations  (Elix  &  Naikar,  2009).  While  the  ADS 
can  be  difficult  to  understand  because  it  treats  human  work  in  an  actor-  and  event- 
independent  way,  ConTA  has  historically  been  easier  to  grasp  because  it  is  only  actor- 
independent,  and  not  event -independent.  The  addition  of  work  situations  to  ConTA 
emphasized  the  event-dependent  nature  of  this  analysis,  which  could  make  this  phase  of 
CWA  even  more  accessible  to  CF  personnel. 

•  Challenges  with  the  Decision  Ladder.  Even  though  ConTA  is  the  most  accessible  of  the 
CWA  techniques,  the  visual  form  of  the  Decision  Ladder  can  still  be  overwhelming  to 
novices.  There  is  potential  to  simplify  this  modelling  tool.8 

•  The  Strategies  Analysis  phase  of  CWA  may  be  difficult  to  apply  in  the  context  of  CF 
procurement.  Strategies  are  artefacts  of  interactions  between  the  work  domain  and  the 
work  support  being  designed.  It  is  likely  that  this  type  of  analysis  is  so  dependent  on  the 
details  of  the  way  that  work  support  is  designed  that  it  will  be  difficult  to  apply  in  the 
context  of  procurement.  Investigating  other  phases  of  CWA  is  likely  to  have  a  higher  return. 

•  Position  of  SOCA  within  CWA.  Since  Vicente’s  (Vicente,  1999)  formulation  of  CWA, 
SOCA  has  been  widely  viewed  as  a  discrete  phase  of  CWA,  but  there  may  be  some  benefit 


8  Gavan  Lintern  is  currently  developing  a  paper  on  this  topic. 
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in  returning  to  Rasmussen’s  (Rasmussen,  et  al.,  1994)  conception  of  it  as  a  phase  running  in 
parallel  with  the  other  four  phases  of  CWA.  The  concept  that  emerged  at  our  meetings  was 
of  dealing  with  high-level  social  organization  issues  during  Work  Domain  Analysis,  and 
then  progressing  through  increasingly  detailed  levels  through  ConTA,  Strategies  Analysis, 
and  Work  Competencies  Analysis. 

•  Non-hierarchical  work  organizations.  Especially  in  the  context  of  network-centric 
warfare,  the  traditionally  hierarchical  structure  of  military  organizations  seems  to  be 
opening  up  to  include  increased  cooperation  between  peers  in  different  organizational  units. 
Currently  many  of  the  changes  in  an  organization’s  structures  evolve  through  interactions 
with  new  systems  and  with  allied  forces  using  similar  systems.  A  constraint-based 
perspective  on  collaboration  may  help  to  inform  the  development  of  team  structures  in  this 
context. 

•  CWA  and  crewing  issues.  CWA  has  already  been  successfully  applied  to  the  development 
of  a  simulation  model  to  test  hypotheses  about  crewing  levels  in  the  context  of  damage 
control  on  a  future  naval  platform  (Coates  &  Cooper,  2009;  Torenvliet,  Coates,  &  Jamieson, 
2008;  Torenvliet  &  Jamieson,  2007;  Torenvliet,  Jamieson,  &  Cournoyer,  2007).  There  is 
potential  to  extend  this  work  so  that  it  serves  as  a  better  starting  point  for  the  application  of 
military  judgment  to  the  setting  of  crewing  levels. 

•  Worker  competencies  analysis  and  CF  Military  Occupation  Codes  (MOCs).  The  CF’s 
MOCs  contain  information  on  the  competencies  that  personnel  in  each  position  in  the  CF 
are  expected  to  possess,  and  so  the  MOCs  form  a  database  of  worker  competencies 
information.  DRDC  has  already  done  some  work  to  understand  how  MOCs  could  be 
integrated  with  network  simulations,  and  there  is  potential  to  expand  this  capability  under 
Project  14dj. 
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3  Research  opportunities 


3.1  Introduction 

The  purpose  of  this  section  is  to  present  the  research  questions  that  were  raised  through  our 
review  of  CF  procurement  (Section  2.2)  and  current  practice  in  requirements  engineering  and 
options  analysis  (Section  2.3),  and  through  our  discussions  of  CWA  (Section  2.4).  This  section 
first  presents  the  questions  and  research  topics  that  were  raised  during  our  discussions,  and  then 
distils  a  number  of  proposals  for  specific  research  projects  from  these  questions. 

3.2  Research  questions 

3.2.1  Related  to  CF  procurement 

Our  review  of  the  CF  procurement  process  has  highlighted  a  number  of  important  questions  in  the 
context  of  Project  14dj.  These  questions  are  listed  below  along  with  the  section  of  the  report  that 
motivates  them. 

1.  Section  2.2.2  -  High-level  overview  of  the  CF  procurement  process  -  How  can  M&S 
techniques  assist  in  the  application  of  the  DNDAF?  DNDAF  views  are  being  promoted  as 
a  common  modelling  language  for  CF  procurement  projects,  especially  projects  that  have  an 
information  technology  component.  There  seems  to  be  an  opportunity  to  supplement  DNDAF 
methods,  especially  at  the  level  of  operational  views,  with  results  and  representations  from 
HF  modelling  techniques.  An  important  question  for  DRDC  in  the  context  of  Project  14dj  is 
whether  there  is  an  opportunity  for  HF  modelling  techniques  to  assist  in  the  application  of  the 
DNDAF. 

2.  Section  2.2.2  -  High-level  overview  of  the  CF  procurement  process  -  How  can  M&S 
techniques  be  tailored  to  an  environment  that  is  perceived  to  be  chronically  under¬ 
staffed  and  under-resourced?  M&S  techniques  are  often  perceived  to  be  costly,  both  in 
terms  of  resources  and  time.  An  important  question  for  DRDC  to  address  in  the  context  of 
Project  14dj  is  just  how  M&S  can  be  used  to  deliver  rapid  results  at  a  level  of  detail  relevant 
to  the  context  in  which  they  are  being  applied. 

3.  Section  2.2.2  -  High-level  overview  of  the  CF  procurement  process  -  How  can  M&S 
techniques  be  tailored  to  provide  risk  reduction?  If  the  current  culture  in  CF  procurement 
is  to  favour  risk  reduction  over  detailed  analysis,  an  important  question  for  DRDC  in  the 
context  of  Project  14dj  is  how  M&S  techniques  can  be  tailored  to  identify  and  analyse  areas 
of  high  risk. 

4.  Section  2.2.2  -  High-level  overview  of  the  CF  procurement  process  -  What  phases  of  the 
procurement  process  can  be  best  addressed  by  M&S  techniques?  The  overview  of  the 
procurement  process  in  this  document  was  informed  by  SME  input  and  a  review  of  a 
collection  of  procurement  documents.  An  important  question  for  DRDC  in  the  context  of 
Project  14dj  is  how  input  from  a  larger  pool  of  SMEs  involved  in  current  or  recent 
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procurement  projects  (e.g.,  JUSTAS,  MHP,  HMCCS,  Fixed-wing  SAR,  etc.)  could  provide 
better  information  as  to  where  the  CF  perceives  problems  in  the  procurement  process.  This 
perspective  could  lead  to  more  robust  insights  about  the  phases  of  procurement  that  can  be 
best  addressed  by  M&S  techniques. 

5.  Section  2.2.2  -  High-level  overview  of  the  CF  procurement  process  -  General  -  How  can 
M&S  techniques  provide  a  context  for  the  application  of  best  military  judgment?  In 

Section  2.2.1,  we  quoted  the  Strategic  Capability  Roadmap  (Chief  of  Force  Development, 
2008b)  as  being  approving  of  the  application  of  best  military  judgment  over  the  use  of  formal 
models.  Immediately  following  that  quote  is  the  following:  “...the  solution  provided  by  the 
software  model  provided  a  starting  point  for  discussion. . .  (p.  52).”  An  important  question  for 
DRDC  in  the  context  of  Project  14dj  is  how  M&S  techniques  can  be  developed  that  provide 
insights  that  can  function  as  the  starting  point  for  discussion. 

6.  Section  2.2.4  -Capability-based  planning  -  How  can  M&S  techniques  assist  in  the 
development  of  scenarios?  The  CBP  process  relies  heavily  on  the  development  of  scenarios 
to  develop  an  understanding  of  future  capability  requirements.  It  already  includes  guidelines 
for  the  development  of  scenarios;  however,  an  important  question  for  DRDC  in  the  context  of 
Project  14dj  is  whether  or  not  there  is  any  potential  for  modelling  techniques  to  increase  the 
quality  of  these  scenarios. 

7.  Section  2.2.5  -Operational  requirements  development  -  How  can  M&S  techniques 
provide  frameworks  to  assist  in  considering  the  full  breadth  of  concerns  when 
developing  operational  requirements?  An  important  question  for  DRDC  in  the  context  of 
Project  14dj  is  to  assess  if  M&S  techniques  can  provide  structure  to  the  development  of 
operational  requirements  that  will  help  to  ensure  that  they  are  based  on  the  full  breadth  of 
applicable  concerns. 

8.  Sections  2.2.5  -  2.2.7  -  Operational  requirements  through  to  technical  requirements  and 
implementation  -  How  can  M&S  techniques  help  to  convert  the  tacit  knowledge  and 
assumptions  behind  requirements  into  explicit  knowledge  in  requirements? 

Requirements  in  one  phase  should  not  be  based  on  tacit  knowledge  and  assumptions  that  may 
not  be  shared  or  understood  by  personnel  interpreting  the  requirements  in  the  next  phase. 
Since  M&S  techniques  typically  make  use  of  naive  investigators  to  characterize  a  work 
context  and  so  help  to  identify  tacit  knowledge  and  assumptions,  an  important  question  for 
DRDC  in  the  context  of  Project  14dj  is  to  assess  whether  formal  modelling  techniques 
applied  by  naive  investigators  can  assist  in  identifying  tacit  knowledge  and  assumptions  for 
better  inclusion  in  requirements. 

9.  Sections  2.2.6-2.2.7  -  Technical  requirements  and  Implementation  -  How  can  M&S 
techniques  help  in  bridging  the  gulf  between  the  problem  space  and  the  solution  space? 

If  it  is  true  that  CF  personnel  develop  requirements  around  a  mental  model  of  an  assumed 
solution,  an  important  question  for  DRDC  in  the  context  of  Project  14dj  is  whether  or  not 
modelling  techniques  can  help  operators  to  externalize  their  mental  model  of  the  assumed 
solution  for  critique  and  correction  by  others,  and  whether  or  not  this  would  assist  in  ensuring 
that  requirements  were  not  unduly  structured  around  an  assumed  solution. 
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10.  Overall  -  How  can  DRDC  become  more  tightly  engaged  in  procurement  projects  to  be 
able  to  advocate  for  appropriate  inclusion?  Outside  of  the  technical  questions  pertaining 
specifically  to  M&S  techniques,  once  DRDC  has  identified  ways  to  have  significant  impact 
on  the  procurement  process  as  per  the  questions  above,  an  important  question  for  DRDC  in 
the  context  of  Project  14dj  will  be  to  investigate  how  to  become  more  tightly  engaged  in 
especially  the  early  phases  of  procurement  projects  so  that  appropriate  research  efforts  can  be 
planned  for  up  front,  rather  than  carried  out  reactively. 

3.2.2  Related  to  the  literature  on  requirements  engineering  and  options 
analysis 

The  following  is  a  list  of  some  potential  research  questions  related  to  the  literature  review  of 

requirements  engineering  and  options  analysis: 

1 1 .  How  can  requirements  be  posed  within  the  important  context  of  why  and  how  (where 
relevant)  without  polluting  the  measurability  and  testability  of  requirements? 

Addressing  this  question  will  ensure  that  any  contributions  of  Project  14dj  to  the  development 
of  requirements  respect  the  criteria  for  requirements  set  by  current  systems  engineering  and 
software  development  practice,  while  at  the  same  time  improving  them. 

12.  How  can  M&S  techniques  support  a  more  efficient  application  of  best  military 
judgment?  If  M&S  techniques  can  be  proposed  that  make  the  jobs  of  operators  and 
engineers  involved  in  requirements  engineering  and  options  analysis  easier,  they  will  be  more 
readily  adopted  by  the  CF. 

13.  How  can  M&S  techniques  support  the  development  of  measurable  and  testable  criteria 
for  human  performance?  M&S  techniques  have  an  inherent  focus  on  Measures  of 
Effectiveness  and  Measures  of  Performance,  and  this  focus  could  assist  in  the  development  of 
human-related  requirements  that  are  posed  in  the  same  language  of  measurability  and 
testability  as  other  system  requirements. 

3.2.3  Related  to  CWA 

The  following  is  a  list  of  some  potential  research  questions  related  to  the  application  of  CWA  to 

requirements  engineering  and  options  analysis  in  CF  procurement: 

14.  Can  WDA  help  to  make  explicit  the  implicit  functional  models  and  assumed 
implementations  of  the  requirements  engineering  process,  and  will  this  help  to  mediate 
the  effect  of  tacit  knowledge  and  unstated  assumptions  on  requirements  development? 

Addressing  this  question  could  help  the  CF  to  develop  requirements  that  do  not  suffer  from 
the  interpretation  challenges  imposed  by  tacit  knowledge  and  unstated  assumptions,  that 
include  a  more  thorough  consideration  of  human  performance  and  decision-making  factors, 
and  that  have  a  greater  likelihood  of  including  consideration  of  all  relevant  factors. 

15.  Can  the  ADS  help  to  provide  requirements  in  their  context,  and  help  to  mediate  the 
effect  of  tacit  knowledge  and  unstated  assumptions  on  requirements  development? 

Including  ADS  models  as  a  common  visualization  to  support  requirements  documentation 
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may  help  to  present  requirements  in  their  context.  When  authoring  or  reading  requirements, 
this  may  help  to  clarify  the  presence  of  tacit  knowledge  or  unstated  assumptions. 

16.  Can  a  tailored  version  of  CWA  and  a  tool  to  support  its  application  be  developed  to 
make  CWA  more  accessible  to  personnel  involved  in  CF  procurement?  Development  of  a 
tailored  version  of  CWA  relevant  to  CF  procurement  and  of  tool  support  for  that  version  of 
CWA  may  be  necessary  preconditions  to  successfully  applying  CWA  to  CF  procurement.  A 
number  of  tools  have  already  been  developed  by  CWA  researchers  to  support  the  application 
of  CWA  (for  example,  the  Work  Domain  Analysis  Workbench  (Sanderson,  Eggleston, 
Skilton,  &  Cameron,  1999)  or  the  CWA  tool  by  researchers  at  Brunei  University9),  and  these 
tools  could  potentially  be  refined  and  extended  for  this  purpose.  Alternatively,  work  to 
resolve  this  question  may  identify  that  a  fully  new  tool  should  be  developed. 

17.  Can  CWA  help  to  identify  and  bound  the  unknown  aspects  of  a  procurement  project? 

One  of  the  major  causes  of  cost  overruns  in  major  Crown  procurement  projects  is  incomplete 
requirements.  If  CWA  can  be  developed  as  a  tool  to  support  the  identification  and  bounding 
of  unknown  aspects  of  a  procurement,  aspects  of  the  requirements  definition  that  are 
incomplete  may  be  identified  earlier.  Even  if  they  cannot  be  resolved,  the  unknowns  can  be 
bounded  and  contingency  resources  can  be  allocated. 

18.  Can  CWA,  and  especially  the  ADS,  assist  in  the  development  of  more  rigorous  scenarios 
in  the  CBP  and  operational  requirements  development  phases?  It  is  not  clear  that  there  is 
currently  any  structured  means  of  assessing  the  completeness  of  the  scenarios  that  are  used 
for  CBP  and  operational  requirements  development.  CWA,  and  especially  the  ADS,  show 
promise  to  support  evaluations  of  scenario  completeness,  and  it  could  be  useful  to  investigate 
this  promise  further. 

19.  Can  ConTA,  with  the  addition  of  consideration  of  work  situations,  be  used  as  a  point-of- 
departure  for  CWA?  If  the  addition  of  work  situations  to  ConTA  increase  the  accessibility 
of  this  phase  of  CWA,  should  ConTA  be  promoted  as  the  starting  point  for  the  application  of 
CWA  to  CF  procurement? 

20.  Can  the  visual  form  of  the  Decision  Ladder  be  improved?  Since  the  visual  form  of  the 
Decision  Ladder  may  be  overwhelming  to  novices,  can  the  visual  form  of  this  modelling  tool 
be  improved  to  make  it  easier  to  read  and  apply? 

21.  How  can  SOCA  be  re-cast  in  parallel  with  the  other  four  phases  of  CWA,  and  is  there 
any  potential  for  this  to  make  the  application  of  CWA  to  CF  procurement  easier  and 
more  relevant?  There  is  an  opportunity  for  some  theory-based  research  (potentially  with  a 
university  partner)  to  revisit  the  position  of  SOCA  within  CWA. 

22.  What  is  the  potential  for  CWA  to  inform  the  development  of  team-structures  for 

systems  in  the  context  of  network-centric  warfare?  SOCA  is  one  of  the  lesser  exercised 
parts  of  CWA,  but  its  potential  for  helping  to  develop  team  structures  could  help  in  the 
developing  ConOps  for  new  systems  that  anticipate  the  effects  of  the  new  systems  on  team 
structures,  and  so  overcome  at  least  a  portion  of  the  task-artefact  cycle. 


9  See  http://www.bmnel.ac.uk/about/acad/sed/sedres/dm/erg/cwa. 
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23.  Can  the  current  technique  for  developing  simulation  models  for  assessments  of  crewing 
levels  be  extended  to  better  support  CF  procurement  questions?  There  is  potential  to 
extend  the  work  done  to  develop  a  network  simulation  of  crewing  issues  (Coates  &  Cooper, 
2009;  Torenvliet,  et  al.,  2008;  Torenvliet  &  Jamieson,  2007;  Torenvliet,  et  al.,  2007)  to  be 
more  useful  within  the  context  of  procurement.  This  could  involve  refining  the  model  to 
allow  it  to  address  a  broader  range  of  crewing  issues,  but  could  also  include  working  with  CF 
sponsors  who  are  involved  in  setting  initial  targets  for  crewing  levels  and  refining  them  as  a 
project  progresses  to  see  how  network-based  simulations  of  crewing  levels  could  be 
developed  that  use  the  data  available  at  each  decision  point. 

24.  Can  the  existing  work  to  integrate  MOCs  with  network  simulations  be  extended,  and 
can  it  be  done  in  the  context  of  the  Worker  Competencies  Analysis  phase  of  CWA? 

Extending  DR  DC's  work  to  integrate  MOCs  with  network  models  could  increase  the 
coherence  between  network  modelling  and  the  fourth  phase  of  CWA. 

3.3  Research  proposals 

The  purpose  of  this  section  is  to  propose  five  specific  research  projects  that  could  be  pursued  by 
Project  14dj.  These  proposals  follow  from  and  consolidate  the  research  questions  documented  in 
Section  3.2.  For  each  proposal,  a  preliminary  research  objective  has  been  developed,  and  links  to 
the  research  questions  motivating  that  research  proposal  are  provided. 

3.3.1  Proposal  1  -  Application  of  CWA  and  M&S  to  the  development  of 
operational  requirements 

Objective.  To  develop  an  approach  to  applying  CWA  and  M&S  to  the  development  of 
operational  requirements  for  CF  procurement. 

Approach.  The  objective  will  be  achieved  by  working  with  an  in-progress  test  procurement 
project  embarking  on  the  phase  of  operational  requirements  development.  The  research  will  be 
conducted  in  the  context  of  the  selected  procurement  project  to  develop  approaches  to  applying 
CWA  and  M&S  techniques  to  assist  in  the  development  of  operational  requirements.  Work  will 
include: 

•  High-level  project  review.  The  SCR  document  from  which  the  selected  procurement 
project  originated  will  be  reviewed  to  gain  an  understanding  of  the  capability  deficiency  and 
how  that  deficiency  has  been  positioned  in  the  context  of  the  future  battle-space. 

•  ADS  model  development.  Appropriate  ADS  models  to  capture  operators’  understanding  of 
the  work  domain  should  be  developed.  Knowledge  elicitation  to  develop  the  ADS  should 
focus  on  identifying  tacit  knowledge  and  unstated  assumptions  so  that  these  types  of 
information  can  be  represented  in  the  models,  as  appropriate.  The  ADS  model  should  be 
aimed  at  the  development  of  system-level  operational  requirements. 

•  Scenario  development.  The  ADS  developed  in  the  previous  work  item  should  be  used  as  a 
basis  for  assessing  the  scenarios  created  by  operational  personnel,  or  as  a  basis  for  assisting 
operational  personnel  in  developing  scenarios.  Attention  should  be  paid  to  methods  for 
using  the  ADS  to  assess  the  breadth  of  scenarios. 
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•  ConTA.  Naikar  et  al.’s  (2006)  guidance  on  the  use  of  work  situations  to  structure  the 
development  of  ConTAs  should  be  applied  to  the  procurement  project  as  relevant  to  help  in 
developing  system-level  operational  requirements.  Knowledge  elicitation  to  develop  the 
Decision  Ladders  should  focus  on  identifying  tacit  knowledge  and  unstated  assumptions  so 
that  these  types  of  information  can  be  represented  in  the  models,  as  appropriate.  New 
research  findings  in  Decision  Ladder  representation  should  be  used  in  this  work,  as  well  as 
any  other  possible  efforts  to  ensure  the  Decision  Ladders  are  represented  in  a  form  that  is 
meaningful  to  CF  personnel. 

•  Requirements  review.  At  the  conclusion  of  these  work  items,  the  findings  and  models  from 
the  previous  work  items  should  be  used  to  conduct  a  review  of  the  actual  operational 
requirements  produced  by  project  staff.  This  review  should  focus  on  identifying  potential 
problems  with  the  requirements  in  terms  of  their  future  use  for  the  development  of  technical 
requirements,  including:  incompleteness,  inconsistencies,  unstated  assumptions,  and 
meaning  built  on  tacit  knowledge.  Alternatives  should  be  posed  that,  while  still  meeting  the 
criteria  of  being  measurable  and  testable,  also  provide  context  for  each  requirement.  If 
possible,  this  review  should  be  conducted  in  cooperation  with  project  staff,  and  findings 
should  be  reviewed  by  project  staff. 

•  Methodological  contribution.  At  each  stage,  the  steps  used  to  develop  the  various  analyses 
should  be  developed  into  an  initial  version  of  a  CWA  manual  for  CF  procurement.  The 
methodological  development  should  focus  on  preparing  a  manual  that  could  be  understood, 
and  potentially  used,  by  CF  personnel.  The  contents  of  the  manual  should  focus  on  the 
development  of  an  efficient  process  that  provides  flexibility  to  identify  and  address  only  the 
most  significant  project  risks.  Comments  should  be  made  on  the  potential  for  CWA 
modelling  to  identify  and  bound  project  unknowns.  Finally,  comments  should  also  be  made 
on  the  form  that  requirements  should  take  if  they  are  to  be  measurable  and  testable  while 
also  providing  appropriate  context. 

Rationale.  This  research  proposal  has  been  developed  around  the  first  three  phases  of  CWA 
because  those  phases  are  the  best  understood  and  show  the  best  potential  for  systematization  in 
the  context  of  CF  procurement.  An  in-progress  project  was  chosen  over  an  already  completed 
project  because  (a)  since  this  work  should  provide  real  benefits  to  a  project  sponsor,  working  with 
an  in-progress  project  should  help  in  securing  funding  for  this  work;  and,  (b)  an  in-progress 
project  is  a  more  realistic  context  than  a  completed  project.  Finally,  the  development  of 
operational  requirements  was  selected  as  a  project  context  because  this  phase  of  procurement  has 
typically  been  found  by  DRDC  scientists  to  be  the  most  amenable  to  research  projects. 

Linkages.  This  research  will  address  the  following  research  questions  from  Section  3.2: 

•  2  -  How  can  M&S  techniques  be  tailored  to  an  environment  that  is  perceived  to  be 
chronically  under-staffed  and  under-resourced? 

•  3  -  How  can  M&S  techniques  be  tailored  to  provide  risk  reduction? 

•  6  -  How  can  M&S  techniques  assist  in  the  development  of  scenarios? 

•  7  -  How  can  M&S  techniques  provide  frameworks  to  assist  in  considering  the  full  breadth 
of  concerns  when  developing  operational  requirements? 
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•  8  -  How  can  M&S  techniques  help  to  convert  the  tacit  knowledge  and  assumptions  behind 
requirements  into  explicit  knowledge  in  requirements? 

•  9  -  Technical  requirements  and  Implementation  -  How  can  M&S  techniques  help  in 
bridging  the  gulf  between  the  problem  space  and  the  solution  space? 

•  1 1  -  How  can  requirements  be  posed  within  the  important  context  of  why  and  how  (where 
relevant)  without  polluting  the  measurability  and  testability  of  requirements? 

•  14  -  Can  WDA  help  to  make  explicit  the  implicit  functional  models  and  assumed 
implementations  of  the  requirements  engineering  process,  and  will  this  help  to  mediate  the 
effect  of  tacit  knowledge  and  unstated  assumptions  on  requirements  development? 

•  15  -  Can  the  ADS  help  to  provide  requirements  in  their  context,  and  help  to  mediate  the 
effect  of  tacit  knowledge  and  unstated  assumptions  on  requirements  development? 

•  17  -  Can  CWA  help  to  identify  and  bound  the  unknown  aspects  of  a  procurement  project? 

•  18  -  Can  CWA,  and  especially  the  ADS,  assist  in  the  development  of  more  rigorous 
scenarios  in  the  CBP  and  operational  requirements  development  phases? 

•  19  -  Can  ConTA,  with  the  addition  of  consideration  of  work  situations,  be  used  as  a  point  - 
of -departure  for  CWA? 

3.3.2  Proposal  2  -  Cognitive  task  analysis  of  requirements  engineering 
and  options  analysis  in  CF  procurement 

Objective.  To  develop  an  understanding  of  the  types  of  M&S  and  HF  support  that  could  be 
developed  to  support  CF  procurement,  by  conducting  a  cognitive  task  analysis  of  requirements 
engineering  and  options  analysis  in  CF  procurement. 

Approach.  The  objective  will  be  achieved  by  working  with  CF  personnel  with  experience  in 
requirements  development  and  options  analysis  during  the  various  phases  of  procurement  to 
understand  how  they  did  their  work,  the  challenges  they  encountered,  the  elements  that  work 
well,  and  the  potential  for  improvements.  Work  will  include: 

•  Review  of  the  phases  of  CF  procurement.  The  phases  of  CF  procurement  should  be 
reviewed  to  determine  the  types  of  CF  personnel  to  interview,  and  the  broad  categories  of 
information  to  elicit.  Ideally,  the  project  team  should  include  ex-CF  personnel  with 
experience  in  all  phases  of  procurement  to  help  in  locating  CF  personnel  to  interview,  to 
help  in  narrowing  down  the  categories  of  information  to  elicit,  and  to  serve  as  a  domain 
interpreter. 

•  Identification  of  interviewees.  The  appropriate  CF  personnel  should  be  identified  and 
contacted  to  arrange  interviews.  We  expect  that  interviews  will  need  to  be  conducted  with 
up  to  seven  different  groups  of  subject  matter  experts  (one  group  with  experience  in  CBP; 
one  group  from  each  environment  with  experience  in  operational  requirements 
development;  and,  one  group  from  each  environment  with  experience  in  technical 
requirements  development). 

•  Development  of  a  knowledge  elicitation  approach.  Once  interviewees  have  been 
identified,  a  detailed  knowledge  elicitation  approach  should  be  developed  for  each  group  of 
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interviewees.  While  we  expect  that  most  interviews  will  need  to  be  conducted  in  a  group,  it 
could  also  be  beneficial  to  conduct  a  more  ethnographic  form  of  knowledge  elicitation,  with 
project  team  members  participating  in  a  series  of  procurement  activities  at  each  phase. 

•  Knowledge  elicitation.  After  the  knowledge  elicitation  approach  has  been  approved,  the 
knowledge  elicitation  activities  should  be  conducted.  Team  debriefs  should  be  held  after 
each  session  with  data  consolidation  sessions  to  be  held  as  appropriate  to  aggregate  results. 

•  Data  reduction  and  validation.  The  data  from  the  interviews  should  be  reduced  into  sets  of 
observations  and  implications.  These  should  be  validated  with  the  CF  personnel  who 
participated  in  the  interviews.  The  data  validation  should  also  include  discussions  with  CF 
personnel  about  their  ideas  on  opportunities  for  reducing  overall  procurement  risk  and  for 
working  more  efficiently,  and  how  M&S  techniques  could  support  this. 

•  Identification  of  opportunities  for  M&S  and  HF  support.  Finally,  the  validated 
observations  and  implications  should  be  reviewed  in  a  workshop  of  interested  parties  to 
determine  the  opportunities  for  M&S  and  HF  support  in  procurement  that  result  from  the 
data. 

Rationale.  This  research  proposal  is  relevant  because  it  will  supplement  the  observations  and 
conclusions  of  this  scoping  study  with  data  from  actual  CF  experiences  with  procurement.  This 
could  help  to  identify  high  priority  opportunities  for  M&S  and  HF  techniques,  especially  to  assist 
in  reducing  program  risk  and  working  more  efficiently  with  scarce  resources. 

Linkages.  This  research  will  address  the  following  research  questions  from  Section  3.2: 

•  2  -  How  can  M&S  techniques  be  tailored  to  an  environment  that  is  perceived  to  be 
chronically  under-staffed  and  under-resourced? 

•  3  -  How  can  M&S  techniques  be  tailored  to  provide  risk  reduction? 

•  4  -  What  phases  of  the  procurement  process  can  be  best  addressed  by  M&S  techniques? 

•  5  -  How  can  M&S  techniques  provide  a  context  for  the  application  of  best  military 
judgment? 

•  10  -  How  can  DRDC  become  more  tightly  engaged  in  procurement  projects  to  be  able  to 
advocate  for  appropriate  inclusion? 

•  12  -  How  can  M&S  techniques  support  a  more  efficient  application  of  best  military 
judgment? 

3.3.3  Proposal  3  -  Development  of  a  CWA  tool  to  support  CF 
procurement 

Objective.  To  develop  the  requirements  for  a  software  tool  to  standardize  the  application  of 
CWA  to  CF  procurement. 

Approach.  This  project  will  follow  from  the  results  of  the  research  proposal  described  in  Section 
3.3.1  ( Proposal  1  —  Application  of  CWA  and  M&S  to  the  development  of  operational 
requirements),  and  will  implement  the  guidance  in  the  CWA  Manual  developed  in  that  research 
as  a  software  tool.  We  expect  that  the  first  phase  of  this  work  will  develop  a  tool  to  support  WDA 
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and  ConTA,  and  subsequent  phases  of  work  will  add  in  support  for  the  remaining  phases  of 
CWA.  This  tool  could  be  developed  as  a  stand-alone  application,  or  as  a  new  module  for  an 
existing  tool.  Work  will  include: 

•  Review  of  the  CWA  manual.  The  CWA  manual  developed  in  the  prior  research  will  be 
reviewed  to  gain  an  understanding  the  modifications  that  it  proposes  to  the  canonical 
methods  of  CWA  in  the  literature. 

•  Review  of  other  CWA  and  M&S  tools.  Existing  CWA  tools  (the  Work  Domain  Analysis 
Workbench  and  the  Brunei  University  CWA  tool)  will  be  reviewed.  This  review  will  note 
their  features,  functionality,  strengths,  and  weaknesses.  In  addition,  the  review  should  also 
include  existing  M&S  tools  currently  held  by  DRDC  or  used  by  the  requirements 
engineering  community,  with  a  view  to  their  suitability  as  a  platform  for  integrating  a  CWA 
tool  for  CF  procurement. 

•  Requirements  development  workshop.  A  workshop  will  be  conducted  to  develop  the 
requirements  for  a  CWA  tool  to  support  CF  procurement.  This  workshop  will  draw  on  the 
results  of  the  previous  reviews,  and  will  aim  to  generate  a  list  of  high-level  requirements  for 
the  tool,  and  will  include  considerations  of  how  this  tool  could  be  integrated  with  other 
current  DRDC  tools  (e.g.,  IPME).  A  final  output  of  the  workshop  will  be  a  comparison 
between  the  high-level  requirements  generated  and  the  existing  CWA  tools  to  develop  a 
recommendation  to  start  tool  development  from  scratch  or  to  extend  an  existing  tool. 

•  Development  of  a  high-level  interface  design.  The  interface  architecture  for  the  tool  will 
be  developed  and  user  tested  with  knowledgeable  subject  matter  experts,  as  possible.  Once 
the  interface  architecture  has  been  developed,  a  paper-based  or  low-fidelity  software-based 
prototype  of  the  overall  application  will  be  developed  for  user  testing  and  redesign. 

•  Development  of  a  high-level  application  architecture.  The  high-level  user  interface 
design  will  be  used  to  develop  a  high-level  application  architecture.  High-level  tradeoffs 
between  the  proposed  interface  design  and  the  application  architecture  will  be  performed  at 
this  time. 

•  (Iterative)  detailed  design  and  implementation.  Once  the  high-level  interface  design  and 
application  architecture  are  approved,  the  detailed  design  and  implementation  will  proceed. 
This  should  be  done  using  an  iterative  process  that  confronts  the  highest-risk  application 
elements  (from  a  design  or  implementation  perspective)  first,  and  that  includes  appropriate 
user  testing  and  redesign. 

The  actual  way  in  which  the  work  items  of  this  project  are  organized  may  depend  on  whether  this 
work  is  conducted  by  DRDC  or  in  cooperation  with  a  third-party  contractor,  and  on  the  level  of 
funding  available.  If  only  a  small  amount  of  initial  funding  can  be  procured,  it  may  be  wise  to 
perform  this  work  in  two  phases  (for  example,  requirements  /  interface  design  and  application 
design  and  implementation). 

Rationale.  This  research  proposal  follows  directly  from  the  objectives  of  Project  14dj. 

Linkages.  This  research  will  address  the  following  research  questions  from  Section  3.2: 

•  16  -  Can  a  tailored  version  of  CWA  and  a  tool  to  support  its  application  be  developed  to 
make  CWA  more  accessible  to  personnel  involved  in  CF  procurement? 
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•  20  -  Can  the  visual  form  of  the  Decision  Ladder  be  improved? 

3.3.4  Proposal  4  -  Applying  SOCA  to  CF  procurement  projects 

Objective.  To  re-engineer  the  SOCA  phase  of  CWA  to  provide  a  framework  that  develops 
requirements  that  specifically  relate  to  organizational  and  teaming  needs. 

Approach.  This  objective  will  be  achieved  by  working  with  a  team  of  DRDC  personnel  and 
CWA  experts  to  analyse  the  most  recent  developments  in  SOCA.  Several  team  and  organizational 
modeling  approaches  will  be  explored  for  their  potential  to  contribute  to  SOCA.  SOCA  as  it 
currently  exists  should  be  challenged  and  reworked  with  a  view  to  substantially  revising  this 
analysis  and  its  positioning  within  the  CWA  framework.  In  the  final  phase  of  this  project,  the 
revision  of  SOCA  should  be  tested  within  a  CF  environment  to  determine  new  technology 
requirements.  These  requirements  should  be  examined  against  existing  CF  procurement 
requirements  to  evaluate  the  feasibility  of  SOCA  to  contribute  to  CF  procurement  projects.  Work 
will  include: 

•  Review  of  SOCA.  The  original  intent  of  SOCA  will  be  reviewed.  As  well  recent  efforts  to 
apply  CWA  to  team  and  social  environments  will  be  reviewed  for  their  potential  role  in  a 
revised  SOCA. 

•  Development  of  a  revised  SOCA.  In  consideration  of  the  intent  of  SOCA,  and  with  current 
tools  for  team  and  social-organizational  analysis,  a  new  SOCA  will  be  developed.  This 
SOCA  may  need  to  be  positioned  differently  within  the  CWA  process  than  the  current 
analysis. 

•  Test  of  the  revised  SOCA.  Within  a  CF  environment  that  has  teaming  and  social- 
organizational  collaboration  needs,  the  revised  SOCA  should  be  field-tested  to  confirm  the 
approach.  The  result  of  this  approach  should  be  a  list  of  social-organizational  and  teaming 
requirements  within  that  environment.  The  revised  SOCA  should  also  be  assessed  for  the 
practicality  of  the  approach  in  terms  of  learnability,  resources  required  and  time  to  generate 
requirements. 

•  Evaluation  against  other  requirements  methods.  To  establish  or  to  challenge  the  value  of 
the  revised  SOCA,  the  requirements  from  stage  3  will  be  compared  against  requirements 
from  an  existing  requirements  exercise  to  determine  whether  the  revised  SOCA  has  the 
potential  to  generate  richer  and  more  useful  teaming  and  organizational  requirements. 

Rationale.  This  research  proposal  is  relevant  because  it  expands  the  tools  available  to  determine 
teaming  and  social-organizational  requirements.  Most  of  the  current  tools  that  examine  teams  or 
organizations  do  not  have  an  engineering  perspective.  As  a  result  they  do  not  address  how  new 
technologies  could  impact  teams  or  organizational  behaviour  and  therefore,  are  not  appropriate 
for  requirements  generation.  With  the  increase  of  network-centric  environments,  team  structure 
requirements  and  social-organizational  requirements  are  critical  to  understand. 

Linkages.  This  research  will  address  the  following  research  questions  from  Section  3.2: 

•  21  -  How  can  social  organization  and  cooperation  analysis  be  re -cast  in  parallel  with  the 
other  four  phases  of  CWA,  and  is  there  any  potential  for  this  to  make  the  application  of 
CWA  to  CF  procurement  easier  and  more  relevant? 
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•  22  -  What  is  the  potential  for  CWA  to  inform  the  development  of  team-structures  for 
systems  in  the  context  of  network-centric  warfare? 

3.3.5  Proposal  5  -  Extension  of  DRDC’s  crewing  effectiveness  network 
model 

Objective.  To  extend  the  DRDC  crewing  effectiveness  network  model  to  better  support  potential 
CF  procurement  questions  around  crewing  levels,  to  develop  this  model  into  a  more  generic 
crewing  effectiveness  tool,  and  to  investigate  if  data  from  the  MOCs  can  be  used  as  a  source  for 
Worker  Competencies  data  in  the  model. 

Approach.  This  work  is  recommended  to  commence  after  the  work  related  to  proposals  3.3.1 
(. Proposal  1  -  Application  of  CWA  and  M&S  to  the  development  of  operational  requirements)  and 
3.3.2  ( Proposal  2  -  Cognitive  task  analysis  of  requirements  engineering  and  options  analysis  in 
CF  procurement )  is  complete,  so  that  insights  especially  about  how  M&S  tools  for  supporting  the 
application  of  best  military  judgment  can  inform  the  work  approach.  Work  items  will  include: 

•  Requirements  investigation  -  general.  The  results  of  the  related  proposals  3.3.1  and  3.3.2 
will  be  reviewed  to  develop  any  insights  from  that  research  as  they  relate  to  the  extension  of 
the  crewing  effectiveness  model.  We  also  recommend  that  interviews  be  held  with  Naval 
Subject  Matter  Experts  to  review  with  them  the  possible  outputs  of  the  crewing 
effectiveness  model  and  to  learn  from  them  their  opinions  of  how  such  models  could 
support  decisions  pertaining  to  crew  size  at  the  various  phases  of  procurement,  and  how 
these  decisions  contribute  to  the  development  of  requirements. 

•  Requirements  investigation  -  integration  of  MOCs.  A  separate  requirements 
development  activity  should  be  conducted  to  propose  ways  that  data  from  the  MOC 
database  could  be  integrated  with  IPME’s  human  performance  model,  to  allow  for  the 
specification  of  crews  for  the  model  that  have  different  capabilities. 

•  Requirements  setting.  A  team  workshop  should  be  held  to  review  the  results  of  the  two 
investigations  and  to  set  the  requirements  for  an  updated  version  of  the  crewing 
effectiveness  model,  and  for  the  development  of  this  model  into  a  more  generic  crewing 
effectiveness  tool. 

•  Implementation  and  testing.  The  tool  will  be  implemented  as  per  the  requirements,  and  a 
test  experiment  will  be  run  to  verify  and  validate  the  types  of  results  that  can  be  produced  by 
the  tool. 

•  Lessons  learned.  Once  the  tool  has  been  implemented  and  tested,  a  lessons  learned 
workshop  should  be  conducted  to  assess  the  appropriateness  of  the  tool  for  supporting 
decision  making  (and  especially  the  application  of  best  military  judgment)  in  the  context  of 
crewing  decisions,  and  the  appropriateness  of  similar  tools  for  supporting  decision  making 
(and  especially  the  application  of  best  military  judgment)  in  the  context  of  military 
procurement  in  general. 

Rationale.  This  project  follows  from  previous  DRDC  research  as  informed  by  the  implications  of 
this  scoping  study,  and  holds  promise  to  demonstrate  how  formal  models  can  support  the 
application  of  best  military  judgment. 
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Linkages.  This  research  will  address  the  following  research  questions  from  Section  3.2: 

•  5  -  How  can  M&S  techniques  provide  a  context  for  the  application  of  best  military 
judgment? 

•  1 3  -  How  can  M&S  techniques  support  the  development  of  measurable  and  testable  criteria 
for  human  performance? 

•  23  -  Can  the  current  technique  for  developing  simulation  models  for  assessments  of  crewing 
levels  be  extended  to  better  support  CF  procurement  questions? 

•  24  -  Can  the  existing  work  to  integrate  MOCs  with  network  simulations  be  extended,  and 
can  it  be  done  in  the  context  of  the  Worker  Competencies  Analysis  phase  of  CWA? 

3.4  Project  sequencing 

Although  the  research  proposals  presented  in  Sections  3.3.1  -  3.3.5  can  each  be  treated  as  stand¬ 
alone  projects,  there  will  be  a  benefit  in  carrying  them  out  in  a  specific  sequence,  as  shown  in 
Figure  9,  below.  If  this  sequence  is  followed,  the  insights  about  the  CF  procurement  process 
gained  from  Proposals  1  and  2  (which  can  be  viewed  as  a  requirements  analysis  phase)  will  be 
able  to  influence  the  methods  chosen  for  the  research  of  Proposals  3  and  5  (which  can  be  viewed 
as  an  implementation  phase). 

We  recommend  that  Proposal  5  be  conducted  at  any  point  in  the  two  years  remaining  in  Project 
14dj.  The  nature  of  this  research  is  such  that  it  would  be  beneficial  to  develop  it  with  a  university 
partner,  and  the  two-year  time-frame  allocated  matches  with  the  typical  duration  of  a  Master’s 
program,  or  with  a  portion  of  a  Doctoral  program. 


April  July  October  January  April  July  October  January  April 

2010  2011 


Figure  9:  Proposed  research  phasing. 
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3.5  Research  question  coverage 

The  research  proposals  presented  in  Sections  3.3.1  -  3.3.5  cover  every  research  question  raised  in 
Section  3.2,  except  for  question  1,  “How  can  M&S  techniques  assist  in  the  application  of  the 
DNDAF?”  While  this  remains  a  relevant  research  question,  it  did  not  fit  easily  into  any  of  the 
research  proposals  we  have  presented. 

If  the  DNDAF  is  a  research  area  that  DRDC  would  like  to  pursue,  we  recommend  that  work  items 
to  more  fully  investigate  challenges  and  opportunities  in  the  application  of  the  DNDAF  (that 
could  potentially  be  resolved  by  M&S  techniques)  be  included  in  Research  Proposal  1  or  2.  Some 
promising  work  has  been  done  to  add  Human  Views  to  the  United  Kingdom’s  counterpart  to 
DNDAF,  the  Ministry  of  Defense  Architectural  Framework  (MODAF)  (Bruseberg,  2008; 
Bruseberg  &  Fletcher,  2008;  Lintern  &  Bruseberg,  2007)  as  well  as  to  architectural  frameworks 
for  NATO  countries  in  general  (Handley  &  Smillie,  2008)  and  Canada  in  particular  (NATO  RTO 
HFM-155  Human  View  Workshop,  2009),  and  it  could  be  useful  to  review  this  work  to 
understand  its  relevance  within  the  context  of  the  DNDAF. 
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4  Conclusions 


In  this  report,  we  have  presented  DRDC  with  an  R&D  roadmap  for  Project  14dj,  “Modelling  and 
Simulation  for  Requirements  Engineering  and  Options  Analysis”  that  includes  five  specific 
research  proposals: 

•  Proposal  1  -  Application  of  CWA  and  M&S  to  the  development  of  operational  requirements 

•  Proposal  2  -  Cognitive  task  analysis  of  requirements  engineering  and  options  analysis  in  CF 
procurement 

•  Proposal  3  -  Development  of  a  CWA  tool  to  support  CF  procurement 

•  Proposal  4  -  Applying  SOCA  to  CF  procurement  projects 

•  Proposal  5  -  Extension  of  DRDC’ s  crewing  effectiveness  network  model 

These  research  proposals  are  based  on  24  research  questions  that  were  derived  from  reviews  of 
the  CF  procurement  process  and  current  requirements  engineering  and  options  analysis  practice, 
and  an  expert  brainstorming  session  on  the  application  of  CWA  to  requirements  engineering  and 
options  analysis.  Each  research  proposal  has  been  presented  with  an  objective  and  an  overview  of 
the  work  items  that  should  be  accomplished  to  reach  the  stated  objectives.  The  research  proposals 
have  been  presented  as  an  overall  research  program  that  includes  an  initial  phase  of  requirements 
analysis  followed  up  by  a  phase  of  modelling  and  tool  implementation. 

Following  the  research  program  we  have  developed  will  provide  DRDC  with  the  following 
benefits: 

•  Understanding  of  the  procurement  process.  Especially  with  the  research  of  Proposals  1 
and  2,  DRDC  will  gain  a  strong  understanding  of  the  CF  procurement  project.  In  addition  to 
the  benefits  for  the  follow-on  research  in  Project  14dj,  this  understanding  could  help  DRDC 
as  a  whole  to  better  target  other  elements  of  its  research  to  have  an  impact  on  the 
procurement  process. 

•  Full  coverage  of  CWA.  The  potential  for  all  five  phases  of  CWA  to  contribute  to  the  CF 
procurement  process  will  be  explored,  which  will  result  in  applied  and  theoretical 
contributions. 

•  Tool  development.  This  research  program  could  provide  DRDC  with  a  tool  for  conducting 
CWA  that  is  integrated  with  other  tools  used  within  the  DRDC  community,  and  with  a  tool 
for  assisting  the  CF  in  determining  crewing  levels  for  future  platforms.  These  are  important 
steps  toward  developing  a  cohesive  M&S  framework  for  DRDC. 

•  Formal  modelling.  Finally,  this  research  program  will  advance  DRDC’s  understanding  of 
how  to  use  formal  modelling  methods  (such  as  IPME)  to  provide  results  that  have  the 
potential  to  inform  the  application  of  best  military  judgment  in  procurement  projects. 

More  broadly,  if  these  research  projects  are  successful,  the  CF  should  benefit  from  an  overall 
reduction  of  risk  in  the  procurement  cycle.  This  should  lead  to  more  predictable  procurement 
projects  that  are  better  able  to  provide  the  CF  with  the  capabilities  required  to  meet  their  strategic 
objectives. 
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Annex  A  Literature  review  of  current  requirements 
engineering  practice 


The  following  pages  reproduce  the  slides  summarizing  the  results  of  the  literature  review  of 
current  requirements  engineering  practice.  These  slides  were  presented  at  the  project  workshop 
held  on  21-22  January  2010. 
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Annex  B  An  overview  of  the  CF  procurement  process 


The  following  pages  reproduce  the  slides  summarizing  the  results  of  the  review  of  the  CF 
procurement  process.  These  slides  were  presented  at  the  project  workshop  held  on  21-22  January 


2010. 
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Overview  of  the  CF  Procurement  Process 
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List  of  symbols/abbreviations/acronyms/initialisms 


ADS.  Abstraction-Decomposition  Space 
AFCS.  Automatic  Flight  Control  System 
ARP.  Applied  Research  Project,  Applied 
Research  Project 
CBP.  Capability  Based  Planning 
CDRL.  Contractor  Data  Requirements  List 
CF.  Canadian  Forces 
CFD.  Chief  of  Force  Development 
ConOps.  Concept  of  Operations 
ConTA.  Control  Tasks  Analysis 
COTS.  Commercial-Off-The-Shelf 
CWA.  Cognitive  Work  Analysis 
DID.  Data  Item  Description 
DND.  Department  of  National  Defence 
DNDAF.  DND/CF  Architectural  Framework 
DRDC.  Defence  Research  &  Development 
Canada 

HF.  Human  Factors 

HLMC.  High-Level  Mandatory  Capability, 
High-Level  Mandatory  Capability 
HSI.  Human  Systems  Integration 
JUSTAS.  Joint  Unmanned  Surveillance 
Target  Acquisition  System 
M&S.  Modelling  &  Simulation,  Modelling 
&  Simulation 

MALE.  Medium  Altitude  Long  Endurance 
MH.  Maritime  Helicopter 


MHRS.  MH  Requirements  Specification 
MOC.  Military  Occupation  Code 
MODAF.  Ministry  of  Defense  Architectural 
Framework 

NASA.  National  Aeronautics  and  Space 
Administration 

PMO.  Project  Management  Office 
PRICIE.  Personnel/Leadership/Individual 
Training,  Research  and 
Development/Operational  Research, 
Infrastructure,  Environment  and 
Organization,  Concepts,  Doctrine, 
Collective  Training,  Information 
Management  &  Technology  &  Equipment 
Support 

R&D.  Research  &  Development 
RFP.  Request  for  Proposal 
SCR.  Strategic  Capability  Roadmap 
SOCA.  Social  Organization  and 
Cooperation  Analysis 

SOR.  Statement  of  Operational  Requirement 
SOW.  Statement  of  Work 
SysML.  Systems  Modeling  Language 
UAV.  Unmanned  Aerial  Vehicle 
UML.  Unified  Modeling  Language 
WDA.  Work-Domain  Analysis 
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